[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:18 dist-test-slave-q8mg krb5kdc[10299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:18 dist-test-slave-q8mg krb5kdc[10299](info): set up 2 sockets
Sep 25 02:31:18 dist-test-slave-q8mg krb5kdc[10299](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 02:31:19.664816 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:19 dist-test-slave-q8mg krb5kdc[10299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:19.792263 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:46801
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:46801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:19.876495 10312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:19.876748 10312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:19.876787 10312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:19.877977 10312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:19.878050 10312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:19.878077 10312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:19.878098 10312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:19.878126 10312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:19.879765 10312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:46801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:46801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:19.880044 10312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:19.880307 10312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:31:19.887145 10312 server_base.cc:1034] running on GCE node
W20260925 02:31:19.887048 10317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:19.887072 10318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:19.887161 10320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:19.887635 10312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:19.887988 10312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:19.889231 10312 hybrid_clock.cc:648] HybridClock initialized: now 1790303479889206 us; error 38 us; skew 500 ppm
Sep 25 02:31:19 dist-test-slave-q8mg krb5kdc[10299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303479, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:19.891927 10312 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:19.892421 10312 webserver.cc:460] Webserver started at http://127.10.13.62:42043/ using document root <none> and password file <none>
I20260925 02:31:19.892678 10312 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:19.892748 10312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:19.892961 10312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:19.894133 10312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "b43422af616b45caa18ade918b02890d"
format_stamp: "Formatted at 2026-09-25 02:31:19 on dist-test-slave-q8mg"
server_key: "028c4045a46c55431e775543ebdafc18"
server_key_iv: "2c06efcee3e0cec9392761bfb404e07c"
server_key_version: "encryptionkey@0"
I20260925 02:31:19.894550 10312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "b43422af616b45caa18ade918b02890d"
format_stamp: "Formatted at 2026-09-25 02:31:19 on dist-test-slave-q8mg"
server_key: "028c4045a46c55431e775543ebdafc18"
server_key_iv: "2c06efcee3e0cec9392761bfb404e07c"
server_key_version: "encryptionkey@0"
I20260925 02:31:19.896004 10312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:19.896914 10327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:19.897138 10312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:19.897226 10312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "b43422af616b45caa18ade918b02890d"
format_stamp: "Formatted at 2026-09-25 02:31:19 on dist-test-slave-q8mg"
server_key: "028c4045a46c55431e775543ebdafc18"
server_key_iv: "2c06efcee3e0cec9392761bfb404e07c"
server_key_version: "encryptionkey@0"
I20260925 02:31:19.897289 10312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:19.917368 10312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:19.919611 10312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:19.919724 10312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:19.926443 10312 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:46801
I20260925 02:31:19.926847 10380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:46801 every 8 connection(s)
I20260925 02:31:19.926909 10312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:19.927657 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 10312
I20260925 02:31:19.927788 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:19.927963 10381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:19.928794 10292 external_mini_cluster.cc:1428] Setting key 28a66a6f8e467f69345d7f69c1f0d632
I20260925 02:31:19.931607 10381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d: Bootstrap starting.
I20260925 02:31:19.932538 10381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:19.933151 10381 log.cc:824] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:19.934048 10381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d: No bootstrap required, opened a new log
I20260925 02:31:19.936759 10381 raft_consensus.cc:348] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43422af616b45caa18ade918b02890d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46801 } }
I20260925 02:31:19.936946 10381 raft_consensus.cc:374] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:19.936975 10381 raft_consensus.cc:729] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43422af616b45caa18ade918b02890d, State: Initialized, Role: FOLLOWER
I20260925 02:31:19.937153 10381 consensus_queue.cc:260] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43422af616b45caa18ade918b02890d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46801 } }
I20260925 02:31:19.937229 10381 raft_consensus.cc:388] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:19.937253 10381 raft_consensus.cc:482] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:19.937322 10381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:19.937940 10381 raft_consensus.cc:504] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43422af616b45caa18ade918b02890d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46801 } }
I20260925 02:31:19.938055 10381 leader_election.cc:302] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b43422af616b45caa18ade918b02890d; no voters:
I20260925 02:31:19.938226 10381 leader_election.cc:288] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:19.938293 10386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:19.938443 10386 raft_consensus.cc:686] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [term 1 LEADER]: Becoming Leader. State: Replica: b43422af616b45caa18ade918b02890d, State: Running, Role: LEADER
I20260925 02:31:19.938493 10381 sys_catalog.cc:564] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:19.938562 10386 consensus_queue.cc:237] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43422af616b45caa18ade918b02890d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46801 } }
I20260925 02:31:19.939088 10388 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b43422af616b45caa18ade918b02890d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43422af616b45caa18ade918b02890d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46801 } } }
I20260925 02:31:19.939074 10387 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b43422af616b45caa18ade918b02890d. Latest consensus state: current_term: 1 leader_uuid: "b43422af616b45caa18ade918b02890d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43422af616b45caa18ade918b02890d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46801 } } }
I20260925 02:31:19.939203 10388 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:19.939237 10387 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:19.939534 10394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:19.940080 10394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:31:19 dist-test-slave-q8mg krb5kdc[10299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:19.943147 10403 mvcc.cc:205] Tried to move back new op lower bound from 7333083053840986112 to 7333083053828677632. Current Snapshot: MvccSnapshot[applied={T|T < 7333083053840986112}]
I20260925 02:31:19.945578 10394 catalog_manager.cc:1347] Generated new cluster ID: 3a89748891e446e6a9e3b68889bee23e
I20260925 02:31:19.945657 10394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:19.949141 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:19.929920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.10.13.62:46801)
0925 02:31:19.930237 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:19.930249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:19.930369 (+ 120us) server_negotiation.cc:407] Connection header received
0925 02:31:19.932634 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:19.932643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:19.932713 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:19.932945 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:19.934621 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:19.936158 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:19.937882 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:19.938492 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:19.938614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:19.943981 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:19.943986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:19.943987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:19.944000 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:19.946663 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:19.947207 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:19.947212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:19.947212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:19.947346 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:19.947681 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:19.947682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:19.947682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:19.948083 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:19.948276 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:19.948585 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:31:19.948764 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":151,"threads_started":1}
I20260925 02:31:19.962803 10394 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:19.963425 10394 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:19.966579 10394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b43422af616b45caa18ade918b02890d: Generated new TSK 0
I20260925 02:31:19.966801 10394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:19.999825 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46801
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:20.092739 10407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:20.093055 10407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:20.093106 10407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:20.094468 10407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:20.094542 10407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:20.094594 10407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:20.096230 10407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:20.096551 10407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:20.096800 10407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:20.103056 10412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:20.103044 10415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:20.103227 10413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:20.103488 10407 server_base.cc:1034] running on GCE node
I20260925 02:31:20.103698 10407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:20.103950 10407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:20.105122 10407 hybrid_clock.cc:648] HybridClock initialized: now 1790303480105109 us; error 44 us; skew 500 ppm
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:20.107810 10407 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:20.108328 10407 webserver.cc:460] Webserver started at http://127.10.13.1:42599/ using document root <none> and password file <none>
I20260925 02:31:20.108556 10407 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:20.108619 10407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:20.108753 10407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:20.109859 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "4b5eb0da2d3c4dab812341534212ab39"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "82677ebcb692ddb1d3f26ffe1b843df1"
server_key_iv: "29af8fc7efa4e8049dc16991398e80aa"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.110222 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "4b5eb0da2d3c4dab812341534212ab39"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "82677ebcb692ddb1d3f26ffe1b843df1"
server_key_iv: "29af8fc7efa4e8049dc16991398e80aa"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.111660 10407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:20.112428 10422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.112614 10407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:20.112712 10407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "4b5eb0da2d3c4dab812341534212ab39"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "82677ebcb692ddb1d3f26ffe1b843df1"
server_key_iv: "29af8fc7efa4e8049dc16991398e80aa"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.112793 10407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:20.138942 10407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:20.141682 10407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:20.141878 10407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:20.142179 10407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:20.142524 10407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:20.142576 10407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.142613 10407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:20.142657 10407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.151547 10407 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:42401
I20260925 02:31:20.151710 10538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:42401 every 8 connection(s)
I20260925 02:31:20.152032 10407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:20.156030 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 10407
I20260925 02:31:20.156136 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:20.156291 10292 external_mini_cluster.cc:1428] Setting key a84d54969cb8f79bf9d845d431ae17db
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:20.163908 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:20.152934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:48179 (local address 127.10.13.62:46801)
0925 02:31:20.153013 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:20.153016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:20.153297 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:31:20.153793 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:20.153794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:20.153805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:20.153859 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:20.155196 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.155728 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.157390 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.157628 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.157697 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:20.160287 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:20.160289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:20.160290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:20.160299 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:20.162246 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.162690 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.162691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.162692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.162769 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.163026 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.163027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.163028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.163209 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:20.163300 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:20.163655 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 02:31:20.163776 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:31:20.163930 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:20.152939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46801 (local address 127.10.13.1:48179)
0925 02:31:20.153170 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:20.153212 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:20.153706 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:20.153915 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:20.153919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:20.154723 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:20.155004 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:20.155010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.155842 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.155843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:20.157295 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:20.157298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.157725 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.157726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:20.157907 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:20.158546 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:20.158558 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:20.160185 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:20.162341 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:20.162349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:20.162351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:20.162605 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:20.162843 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:20.162844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:20.162844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:20.162949 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:20.163338 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:20.163346 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:20.163491 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 02:31:20.163597 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:31:20.163746 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":124,"threads_started":1}
I20260925 02:31:20.164737 10539 heartbeater.cc:347] Connected to a master server at 127.10.13.62:46801
I20260925 02:31:20.164908 10539 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:20.165138 10539 heartbeater.cc:511] Master 127.10.13.62:46801 requested a full tablet report, sending...
I20260925 02:31:20.165971 10345 ts_manager.cc:194] Registered new tserver with Master: 4b5eb0da2d3c4dab812341534212ab39 (127.10.13.1:42401)
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
I20260925 02:31:20.167104 10345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:48179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:20.188095 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46801
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:20.283330 10544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:20.283527 10544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:20.283551 10544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:20.284765 10544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:20.284806 10544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:20.284833 10544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:20.286415 10544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:20.286691 10544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:20.286938 10544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:20.292196 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:20.292171 10549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:20.292369 10544 server_base.cc:1034] running on GCE node
W20260925 02:31:20.292443 10550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:20.292647 10544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:20.292946 10544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:20.294169 10544 hybrid_clock.cc:648] HybridClock initialized: now 1790303480294152 us; error 26 us; skew 500 ppm
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:20.297194 10544 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:20.297742 10544 webserver.cc:460] Webserver started at http://127.10.13.2:39997/ using document root <none> and password file <none>
I20260925 02:31:20.298025 10544 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:20.298102 10544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:20.298265 10544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:20.299542 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "69bb5c235a3c42e185f0c7861f4f2fc5"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "e33660f120c790891ebd81fd3862ec59"
server_key_iv: "fedb3d4fc3f8bec85b4c9e6a28921408"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.299980 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "69bb5c235a3c42e185f0c7861f4f2fc5"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "e33660f120c790891ebd81fd3862ec59"
server_key_iv: "fedb3d4fc3f8bec85b4c9e6a28921408"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.301786 10544 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 02:31:20.302811 10559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.303117 10544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:20.303241 10544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "69bb5c235a3c42e185f0c7861f4f2fc5"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "e33660f120c790891ebd81fd3862ec59"
server_key_iv: "fedb3d4fc3f8bec85b4c9e6a28921408"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.303325 10544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:20.314465 10544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:20.316944 10544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:20.317122 10544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:20.317382 10544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:20.317725 10544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:20.317786 10544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.317834 10544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:20.317862 10544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.325426 10544 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:36905
I20260925 02:31:20.325469 10676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:36905 every 8 connection(s)
I20260925 02:31:20.325847 10544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:20.334048 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 10544
I20260925 02:31:20.334149 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:20.334333 10292 external_mini_cluster.cc:1428] Setting key c91c4adb0aedbaa33497abd71248c673
I20260925 02:31:20.335762 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:20.326698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:56257 (local address 127.10.13.62:46801)
0925 02:31:20.326789 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:20.326792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:20.327006 (+ 214us) server_negotiation.cc:407] Connection header received
0925 02:31:20.327498 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:20.327499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:20.327511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:20.327575 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:20.328587 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.329289 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.330259 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.330465 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.330530 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:20.332655 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:20.332656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:20.332657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:20.332664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:20.334036 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.334603 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.334604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.334605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.334698 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.335005 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.335006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.335007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.335169 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:20.335254 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:20.335567 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 02:31:20.335660 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:31:20.335996 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:20.326652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46801 (local address 127.10.13.2:56257)
0925 02:31:20.326844 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:20.326882 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:20.327407 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:20.327626 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:20.327630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:20.328276 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:20.328488 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:20.328493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.329375 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.329376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:20.330169 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:20.330170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.330546 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.330547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:20.330649 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:20.331211 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:20.331220 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:20.332567 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:20.334139 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:20.334147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:20.334151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:20.334502 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:20.334773 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:20.334774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:20.334774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:20.334884 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:20.335259 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:20.335267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:20.335418 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 02:31:20.335542 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 02:31:20.335719 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
I20260925 02:31:20.336557 10677 heartbeater.cc:347] Connected to a master server at 127.10.13.62:46801
I20260925 02:31:20.336683 10677 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:20.336930 10677 heartbeater.cc:511] Master 127.10.13.62:46801 requested a full tablet report, sending...
I20260925 02:31:20.337392 10345 ts_manager.cc:194] Registered new tserver with Master: 69bb5c235a3c42e185f0c7861f4f2fc5 (127.10.13.2:36905)
I20260925 02:31:20.338038 10345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:56257
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:20.363716 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46801
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:20.456612 10682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:20.456858 10682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:20.456929 10682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:20.458227 10682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:20.458290 10682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:20.458338 10682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:20.459821 10682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:20.460101 10682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:20.460337 10682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:20.466317 10690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:20.466393 10688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:20.466714 10682 server_base.cc:1034] running on GCE node
W20260925 02:31:20.467108 10687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:20.467331 10682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:20.467546 10682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:20.468761 10682 hybrid_clock.cc:648] HybridClock initialized: now 1790303480468748 us; error 41 us; skew 500 ppm
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:20.471282 10682 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:20.471726 10682 webserver.cc:460] Webserver started at http://127.10.13.3:41989/ using document root <none> and password file <none>
I20260925 02:31:20.471984 10682 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:20.472062 10682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:20.472208 10682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:20.473363 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "05e81e49f60c44a4a3585f942fba0e53"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "0ad7896e5fbd0a59436fb2516762a107"
server_key_iv: "67d34107af3d46eeca015f80b869275c"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.473733 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "05e81e49f60c44a4a3585f942fba0e53"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "0ad7896e5fbd0a59436fb2516762a107"
server_key_iv: "67d34107af3d46eeca015f80b869275c"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.475167 10682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:20.475984 10697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.476209 10682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:20.476320 10682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "05e81e49f60c44a4a3585f942fba0e53"
format_stamp: "Formatted at 2026-09-25 02:31:20 on dist-test-slave-q8mg"
server_key: "0ad7896e5fbd0a59436fb2516762a107"
server_key_iv: "67d34107af3d46eeca015f80b869275c"
server_key_version: "encryptionkey@0"
I20260925 02:31:20.476397 10682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:20.498227 10682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:20.500445 10682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:20.500624 10682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:20.500927 10682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:20.501279 10682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:20.501336 10682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.501380 10682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:20.501407 10682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:20.509683 10682 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:43081
I20260925 02:31:20.509837 10811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:43081 every 8 connection(s)
I20260925 02:31:20.510128 10682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:20.519802 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 10682
I20260925 02:31:20.519884 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:20.519925 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:20.511079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:60265 (local address 127.10.13.62:46801)
0925 02:31:20.511158 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:20.511161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:20.511485 (+ 324us) server_negotiation.cc:407] Connection header received
0925 02:31:20.511940 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:20.511941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:20.511952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:20.512010 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:20.513022 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.513554 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.514622 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.514823 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.514920 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:20.517108 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:20.517109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:20.517110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:20.517117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:20.518602 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.518969 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.518970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.518971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.519032 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.519290 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.519292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.519292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.519416 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:20.519479 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:20.519730 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 02:31:20.519815 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:31:20.520069 10292 external_mini_cluster.cc:1428] Setting key 20fda344759720736945987b4d488b2d
I20260925 02:31:20.520077 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:20.511088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46801 (local address 127.10.13.3:60265)
0925 02:31:20.511351 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:20.511390 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:20.511853 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:20.512068 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:20.512073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:20.512646 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:20.512839 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:20.512843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.513638 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.513639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:20.514501 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:20.514503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.514956 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.514957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:20.515118 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:20.515574 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:20.515583 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:20.516996 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:20.518681 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:20.518687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:20.518689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:20.518897 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:20.519108 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:20.519108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:20.519108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:20.519204 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:20.519482 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:20.519487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:20.519600 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 02:31:20.519681 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:31:20.519795 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":159,"threads_started":1}
I20260925 02:31:20.520737 10812 heartbeater.cc:347] Connected to a master server at 127.10.13.62:46801
I20260925 02:31:20.520846 10812 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:20.521077 10812 heartbeater.cc:511] Master 127.10.13.62:46801 requested a full tablet report, sending...
I20260925 02:31:20.521503 10344 ts_manager.cc:194] Registered new tserver with Master: 05e81e49f60c44a4a3585f942fba0e53 (127.10.13.3:43081)
I20260925 02:31:20.521893 10344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:60265
I20260925 02:31:20.532603 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:20.557557 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:20.548801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.10.13.1:42401)
0925 02:31:20.549113 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:20.549118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:20.549130 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:31:20.549157 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:20.549159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:20.549174 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:20.549248 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:20.549810 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.550628 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.551366 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.551703 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.551761 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:20.554182 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:20.554190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:20.554190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:20.554198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:20.555874 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.556234 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.556238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.556238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.556300 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.556704 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.556705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.556706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.556858 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:20.557277 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:20.557319 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 02:31:20.557387 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":141,"threads_started":1}
W20260925 02:31:20.557986 10451 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49104
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:20.582620 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:20.574691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.10.13.1:42401)
0925 02:31:20.574775 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:20.574777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:20.574832 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:31:20.574948 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:20.574949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:20.574975 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:20.575040 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:20.575713 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.576287 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.577244 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:20.577461 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:20.577526 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:20.580013 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:20.580015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:20.580015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:20.580025 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:20.581618 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.581960 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.581961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.581961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.582033 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:20.582195 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:20.582195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:20.582195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:20.582287 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:20.582397 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:20.582480 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 02:31:20.582552 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:31:20.583261 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 10407
I20260925 02:31:20.590565 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 10544
I20260925 02:31:20.596031 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 10682
I20260925 02:31:20.600855 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 10312
2026-09-25T02:31:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2013 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10826](info): set up 2 sockets
Sep 25 02:31:20 dist-test-slave-q8mg krb5kdc[10826](info): commencing operation
krb5kdc: starting...
W20260925 02:31:21.643532 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:21 dist-test-slave-q8mg krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:21.764751 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:44219
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:21.848335 10840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:21.848556 10840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:21.848579 10840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:21.849740 10840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:21.849782 10840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:21.849797 10840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:21.849809 10840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:21.849821 10840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:21.851264 10840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:21.851472 10840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:21.851646 10840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:21.856528 10848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:21.856536 10846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:21.856635 10840 server_base.cc:1034] running on GCE node
W20260925 02:31:21.856756 10845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:21.857035 10840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:21.857347 10840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:21.858515 10840 hybrid_clock.cc:648] HybridClock initialized: now 1790303481858499 us; error 33 us; skew 500 ppm
Sep 25 02:31:21 dist-test-slave-q8mg krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:21.861279 10840 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:21.861676 10840 webserver.cc:460] Webserver started at http://127.10.13.62:43383/ using document root <none> and password file <none>
I20260925 02:31:21.861889 10840 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:21.861931 10840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:21.862039 10840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:21.863108 10840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "8561e64c670443a79600bfbb1098c2bf"
format_stamp: "Formatted at 2026-09-25 02:31:21 on dist-test-slave-q8mg"
server_key: "010beb755edf0a6deb87ba51ff3feeaa"
server_key_iv: "18b44eef78c95ff402282bf1a55678ad"
server_key_version: "encryptionkey@0"
I20260925 02:31:21.863423 10840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "8561e64c670443a79600bfbb1098c2bf"
format_stamp: "Formatted at 2026-09-25 02:31:21 on dist-test-slave-q8mg"
server_key: "010beb755edf0a6deb87ba51ff3feeaa"
server_key_iv: "18b44eef78c95ff402282bf1a55678ad"
server_key_version: "encryptionkey@0"
I20260925 02:31:21.864732 10840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:31:21.865604 10855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:21.865775 10840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:21.865854 10840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "8561e64c670443a79600bfbb1098c2bf"
format_stamp: "Formatted at 2026-09-25 02:31:21 on dist-test-slave-q8mg"
server_key: "010beb755edf0a6deb87ba51ff3feeaa"
server_key_iv: "18b44eef78c95ff402282bf1a55678ad"
server_key_version: "encryptionkey@0"
I20260925 02:31:21.865908 10840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:21.882313 10840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:21.884519 10840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:21.884644 10840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:21.888597 10840 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:44219
I20260925 02:31:21.888659 10907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:44219 every 8 connection(s)
I20260925 02:31:21.889086 10840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:21.890022 10908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:21.890045 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 10840
I20260925 02:31:21.890141 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:21.890321 10292 external_mini_cluster.cc:1428] Setting key 2b21c15f74f52047c1ad907bd515c480
I20260925 02:31:21.893586 10908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf: Bootstrap starting.
I20260925 02:31:21.894333 10908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:21.894636 10908 log.cc:824] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:21.895466 10908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf: No bootstrap required, opened a new log
Sep 25 02:31:21 dist-test-slave-q8mg krb5kdc[10826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:21.897629 10908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8561e64c670443a79600bfbb1098c2bf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44219 } }
I20260925 02:31:21.897827 10908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:21.897876 10908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8561e64c670443a79600bfbb1098c2bf, State: Initialized, Role: FOLLOWER
I20260925 02:31:21.898020 10908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8561e64c670443a79600bfbb1098c2bf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44219 } }
I20260925 02:31:21.898123 10908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:21.898182 10908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:21.898232 10908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:21.898960 10908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8561e64c670443a79600bfbb1098c2bf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44219 } }
I20260925 02:31:21.899092 10908 leader_election.cc:302] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8561e64c670443a79600bfbb1098c2bf; no voters:
I20260925 02:31:21.899290 10908 leader_election.cc:288] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:21.899427 10913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:21.899590 10908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:21.899639 10913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [term 1 LEADER]: Becoming Leader. State: Replica: 8561e64c670443a79600bfbb1098c2bf, State: Running, Role: LEADER
I20260925 02:31:21.899865 10913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8561e64c670443a79600bfbb1098c2bf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44219 } }
I20260925 02:31:21.900252 10914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8561e64c670443a79600bfbb1098c2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8561e64c670443a79600bfbb1098c2bf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44219 } } }
I20260925 02:31:21.900334 10914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:21.900511 10915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8561e64c670443a79600bfbb1098c2bf. Latest consensus state: current_term: 1 leader_uuid: "8561e64c670443a79600bfbb1098c2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8561e64c670443a79600bfbb1098c2bf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44219 } } }
I20260925 02:31:21.900583 10915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:21.901186 10923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:21.901748 10923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:21.902082 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:21.890894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.10.13.62:44219)
0925 02:31:21.891113 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:21.891123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:21.891159 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:31:21.891721 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:21.891731 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:21.891799 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:21.891959 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:21.892623 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:21.894047 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:21.894873 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:21.895418 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:21.895531 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:21.898218 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:21.898223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:21.898224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:21.898240 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:21.900371 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:21.900855 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:21.900858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:21.900859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:21.900992 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:21.901212 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:21.901213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:21.901214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:21.901492 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:21.901676 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:21.901776 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:31:21.901915 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260925 02:31:21.903419 10923 catalog_manager.cc:1347] Generated new cluster ID: c338a158410d470e8283e51f0657e96b
I20260925 02:31:21.903461 10923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:21.912734 10923 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:21.913339 10923 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:21.917239 10923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8561e64c670443a79600bfbb1098c2bf: Generated new TSK 0
I20260925 02:31:21.917435 10923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:21.951926 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44219
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:22.036446 10935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:22.036720 10935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:22.036747 10935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:22.037961 10935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:22.038004 10935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:22.038031 10935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:22.039522 10935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:22.039739 10935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:22.039930 10935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:22.045240 10940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:22.045214 10943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:22.045287 10941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:22.045410 10935 server_base.cc:1034] running on GCE node
I20260925 02:31:22.045796 10935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:22.046083 10935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:22.047243 10935 hybrid_clock.cc:648] HybridClock initialized: now 1790303482047230 us; error 33 us; skew 500 ppm
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:22.050079 10935 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:22.050577 10935 webserver.cc:460] Webserver started at http://127.10.13.1:42785/ using document root <none> and password file <none>
I20260925 02:31:22.050846 10935 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:22.050900 10935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:22.051083 10935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:22.052201 10935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "ff6759a4f234443ea06ea3666d718d01"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "da0d0a27290d7362b5f4963a47e03b6f"
server_key_iv: "93e419f68baaa8438cd5e1e272148b98"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.052573 10935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "ff6759a4f234443ea06ea3666d718d01"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "da0d0a27290d7362b5f4963a47e03b6f"
server_key_iv: "93e419f68baaa8438cd5e1e272148b98"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.054148 10935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 02:31:22.054993 10950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.055269 10935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:22.055387 10935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "ff6759a4f234443ea06ea3666d718d01"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "da0d0a27290d7362b5f4963a47e03b6f"
server_key_iv: "93e419f68baaa8438cd5e1e272148b98"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.055472 10935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:22.075738 10935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:22.077850 10935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:22.078030 10935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:22.078300 10935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:22.078634 10935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:22.078689 10935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.078725 10935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:22.078773 10935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.084805 10935 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:37007
I20260925 02:31:22.084857 11065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:37007 every 8 connection(s)
I20260925 02:31:22.085265 10935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:22.087294 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 10935
I20260925 02:31:22.087407 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:22.087584 10292 external_mini_cluster.cc:1428] Setting key f027200d032759489fdebc106dca1145
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:22.096504 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.086091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:47825 (local address 127.10.13.62:44219)
0925 02:31:22.086181 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.086184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.086393 (+ 209us) server_negotiation.cc:407] Connection header received
0925 02:31:22.087384 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.087386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.087402 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.087474 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:22.088416 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.089213 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.090317 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.090663 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.090750 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.093101 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:22.093103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:22.093104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:22.093113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:22.094928 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.095370 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.095371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.095372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.095432 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.095703 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.095705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.095705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.095862 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:22.095944 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.096295 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.096403 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
I20260925 02:31:22.096626 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.086044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44219 (local address 127.10.13.1:47825)
0925 02:31:22.086255 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:22.086298 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:22.087257 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:22.087554 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:22.087559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:22.088000 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:22.088308 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.088315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.089330 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.089332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.090224 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.090227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.090760 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.090761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.090884 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.091457 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:22.091466 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:22.093005 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:22.095033 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:22.095040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:22.095042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:22.095287 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:22.095517 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:22.095518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:22.095518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:22.095629 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:22.095963 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:22.095971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:22.096141 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 02:31:22.096242 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:31:22.096385 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":110,"threads_started":1}
I20260925 02:31:22.097262 11066 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44219
I20260925 02:31:22.097369 11066 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:22.097525 11066 heartbeater.cc:511] Master 127.10.13.62:44219 requested a full tablet report, sending...
I20260925 02:31:22.098106 10872 ts_manager.cc:194] Registered new tserver with Master: ff6759a4f234443ea06ea3666d718d01 (127.10.13.1:37007)
I20260925 02:31:22.098693 10872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:47825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:22.116583 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44219
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:22.201764 11071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:22.201992 11071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:22.202023 11071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:22.203233 11071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:22.203279 11071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:22.203306 11071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:22.204911 11071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:22.205180 11071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:22.205449 11071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:22.210711 11076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:22.210716 11077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:22.210721 11079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:22.210986 11071 server_base.cc:1034] running on GCE node
I20260925 02:31:22.211295 11071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:22.211549 11071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:22.212718 11071 hybrid_clock.cc:648] HybridClock initialized: now 1790303482212715 us; error 42 us; skew 500 ppm
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:22.215417 11071 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:22.215899 11071 webserver.cc:460] Webserver started at http://127.10.13.2:33701/ using document root <none> and password file <none>
I20260925 02:31:22.216132 11071 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:22.216176 11071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:22.216275 11071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:22.217391 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "67d856ffd0164cbbac3f26b28cb0bfa0"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "5479f7c07d9b2a724325dd8d937aa3b9"
server_key_iv: "16d2289d6b93ad6026c97cceb8832d9d"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.217713 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "67d856ffd0164cbbac3f26b28cb0bfa0"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "5479f7c07d9b2a724325dd8d937aa3b9"
server_key_iv: "16d2289d6b93ad6026c97cceb8832d9d"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.219103 11071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:22.219866 11086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.220065 11071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:22.220145 11071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "67d856ffd0164cbbac3f26b28cb0bfa0"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "5479f7c07d9b2a724325dd8d937aa3b9"
server_key_iv: "16d2289d6b93ad6026c97cceb8832d9d"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.220212 11071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:22.240597 11071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:22.242911 11071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:22.243033 11071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:22.243259 11071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:22.243539 11071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:22.243572 11071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.243595 11071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:22.243607 11071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.249325 11071 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:40761
I20260925 02:31:22.249416 11200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:40761 every 8 connection(s)
I20260925 02:31:22.249740 11071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:22.252120 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 11071
I20260925 02:31:22.252202 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:22.252373 10292 external_mini_cluster.cc:1428] Setting key 7e53ddea57b10058690ff7a7b9508993
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:22.260403 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.250546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:37985 (local address 127.10.13.62:44219)
0925 02:31:22.250606 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.250608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.250824 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:31:22.251748 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.251749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.251760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.251814 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:22.252594 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.253325 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.254462 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.254686 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.254743 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.257101 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:22.257103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:22.257103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:22.257112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:22.258876 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.259322 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.259323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.259324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.259388 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.259666 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.259668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.259668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.259827 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:22.259906 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.260206 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.260305 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:31:22.260447 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.250492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44219 (local address 127.10.13.2:37985)
0925 02:31:22.250681 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:22.250725 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:22.251656 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:22.251879 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:22.251883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:22.252194 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:22.252476 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.252483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.253429 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.253430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.254353 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.254357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.254773 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.254774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.254926 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.255477 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:22.255489 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:22.257001 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:22.258976 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:22.258984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:22.258986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:22.259238 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:22.259481 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:22.259482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:22.259482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:22.259592 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:22.259902 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:22.259908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:22.260059 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 02:31:22.260147 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:31:22.260275 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260925 02:31:22.261101 11201 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44219
I20260925 02:31:22.261222 11201 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
I20260925 02:31:22.261407 11201 heartbeater.cc:511] Master 127.10.13.62:44219 requested a full tablet report, sending...
I20260925 02:31:22.261845 10872 ts_manager.cc:194] Registered new tserver with Master: 67d856ffd0164cbbac3f26b28cb0bfa0 (127.10.13.2:40761)
I20260925 02:31:22.262239 10872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:37985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:22.281032 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44219
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:22.364843 11206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:22.365072 11206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:22.365097 11206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:22.366307 11206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:22.366350 11206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:22.366377 11206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:22.367810 11206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:22.368007 11206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:22.368188 11206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:22.373565 11211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:22.373638 11206 server_base.cc:1034] running on GCE node
W20260925 02:31:22.373588 11214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:22.373574 11212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:22.374055 11206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:22.374269 11206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:22.375415 11206 hybrid_clock.cc:648] HybridClock initialized: now 1790303482375385 us; error 49 us; skew 500 ppm
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:22.377987 11206 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:22.378494 11206 webserver.cc:460] Webserver started at http://127.10.13.3:44867/ using document root <none> and password file <none>
I20260925 02:31:22.378743 11206 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:22.378811 11206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:22.378959 11206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:22.380023 11206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "49e4dcff793f40e7b95622082036bd2b"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "70986b39f05acde85549ae8558ff7cf8"
server_key_iv: "1615995ad7e4651f765a6ce0510b1723"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.380421 11206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "49e4dcff793f40e7b95622082036bd2b"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "70986b39f05acde85549ae8558ff7cf8"
server_key_iv: "1615995ad7e4651f765a6ce0510b1723"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.381850 11206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:31:22.382611 11221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.382787 11206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:22.382891 11206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "49e4dcff793f40e7b95622082036bd2b"
format_stamp: "Formatted at 2026-09-25 02:31:22 on dist-test-slave-q8mg"
server_key: "70986b39f05acde85549ae8558ff7cf8"
server_key_iv: "1615995ad7e4651f765a6ce0510b1723"
server_key_version: "encryptionkey@0"
I20260925 02:31:22.382973 11206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:22.403905 11206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:22.406317 11206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:22.406528 11206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:22.406800 11206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:22.407177 11206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:22.407240 11206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.407286 11206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:22.407315 11206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:22.413506 11206 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:35045
I20260925 02:31:22.413544 11335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:35045 every 8 connection(s)
I20260925 02:31:22.413875 11206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:22.416285 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 11206
I20260925 02:31:22.416392 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:22.416569 10292 external_mini_cluster.cc:1428] Setting key 5ab24113da70e7c27f6384af72d556d2
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:22.424757 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.414692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:58487 (local address 127.10.13.62:44219)
0925 02:31:22.414764 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.414768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.415071 (+ 303us) server_negotiation.cc:407] Connection header received
0925 02:31:22.416181 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.416183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.416199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.416285 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:22.417122 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.417803 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.418846 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.419064 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.419160 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.421625 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:22.421628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:22.421628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:22.421637 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:22.423103 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.423584 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.423585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.423586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.423654 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.423939 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.423940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.423941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.424165 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:22.424235 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.424567 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.424654 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:31:22.425043 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.414680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44219 (local address 127.10.13.3:58487)
0925 02:31:22.414927 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:22.414966 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:22.416036 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:22.416341 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:22.416345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:22.416728 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:22.417020 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.417027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.417919 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.417920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.418752 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.418754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.419171 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.419172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.419307 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.419851 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:22.419863 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:22.421508 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:22.423214 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:22.423222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:22.423224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:22.423494 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:22.423736 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:22.423737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:22.423738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:22.423861 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:22.424238 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:22.424246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:22.424400 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 02:31:22.424530 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:31:22.424709 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":130,"threads_started":1}
I20260925 02:31:22.425804 11336 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44219
I20260925 02:31:22.425959 11336 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:22.426205 11336 heartbeater.cc:511] Master 127.10.13.62:44219 requested a full tablet report, sending...
I20260925 02:31:22.426630 10872 ts_manager.cc:194] Registered new tserver with Master: 49e4dcff793f40e7b95622082036bd2b (127.10.13.3:35045)
I20260925 02:31:22.427137 10872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:58487
I20260925 02:31:22.428341 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:31:22.436378 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.430129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.10.13.62:44219)
0925 02:31:22.430204 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.430206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.430316 (+ 110us) server_negotiation.cc:407] Connection header received
0925 02:31:22.430369 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.430370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.430380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.430435 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:22.431066 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.431537 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.432318 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.432501 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.432567 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.433796 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:22.433797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:22.433798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:22.433804 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:22.435220 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.435550 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.435550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.435551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.435606 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.435778 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.435779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.435779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.435928 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:22.436061 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.436246 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.436312 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:31:22.439049 10872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:22.440689 10872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:22.447901 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.443425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:35045 (local address 127.0.0.1:46092)
0925 02:31:22.443686 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:22.443695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:22.443734 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:22.444228 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:22.444231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:22.444233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:22.444311 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.444313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.445755 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.445756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.446672 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.446675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.447578 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.447580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.447660 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.447674 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:31:22.447750 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:31:22.447813 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":38,"thread_start_us":111,"threads_started":1}
I20260925 02:31:22.447932 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.443729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.10.13.3:35045)
0925 02:31:22.443994 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.444011 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.444024 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:31:22.444086 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.444089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.444107 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.444195 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:22.444404 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.445646 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.446776 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.447402 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.447471 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.447542 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.447784 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.447829 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":85,"threads_started":1}
I20260925 02:31:22.448101 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.443424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37007 (local address 127.0.0.1:41920)
0925 02:31:22.443615 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:22.443658 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:22.443732 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:22.444540 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:22.444541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:22.444544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:22.444598 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.444599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.445700 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.445701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.446966 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.446969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.447859 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.447860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.447927 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.447931 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:22.447961 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 02:31:22.448019 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260925 02:31:22.448916 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.443424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:40761 (local address 127.0.0.1:43188)
0925 02:31:22.443857 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:22.443865 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:22.443900 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:22.444370 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:22.444371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:22.444373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:22.444459 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.444462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.446155 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.446156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.447173 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.447175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.448614 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.448615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.448685 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.448690 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:22.448760 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:31:22.448803 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":4,"thread_start_us":111,"threads_started":1}
I20260925 02:31:22.448782 11270 tablet_service.cc:1467] Processing CreateTablet for tablet da93521f6b6b4ae2926cff09e885dcb3 (DEFAULT_TABLE table=test-table [id=92c93efd3e4348328f6f2577963fa36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:22.448932 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.443952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.10.13.2:40761)
0925 02:31:22.444198 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.444203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.444213 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:22.444249 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.444251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.444266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.444352 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:22.444721 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.446047 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.447306 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.448510 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.448613 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.448678 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.448776 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.448818 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":137,"threads_started":1}
I20260925 02:31:22.449219 11270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da93521f6b6b4ae2926cff09e885dcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:22.449704 11134 tablet_service.cc:1467] Processing CreateTablet for tablet da93521f6b6b4ae2926cff09e885dcb3 (DEFAULT_TABLE table=test-table [id=92c93efd3e4348328f6f2577963fa36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:22.450080 11134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da93521f6b6b4ae2926cff09e885dcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:22.452256 11353 tablet_bootstrap.cc:492] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b: Bootstrap starting.
I20260925 02:31:22.452414 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.443723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.10.13.1:37007)
0925 02:31:22.444337 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.444342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.444353 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:22.444394 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.444396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.444412 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.444535 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:22.444688 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.445586 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.447061 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.447771 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.448281 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.448351 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.448376 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.452326 (+ 3950us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":100,"threads_started":1}
I20260925 02:31:22.453099 11353 tablet_bootstrap.cc:654] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:22.453410 10999 tablet_service.cc:1467] Processing CreateTablet for tablet da93521f6b6b4ae2926cff09e885dcb3 (DEFAULT_TABLE table=test-table [id=92c93efd3e4348328f6f2577963fa36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:22.453792 11353 log.cc:824] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:22.453802 10999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da93521f6b6b4ae2926cff09e885dcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:22.454795 11353 tablet_bootstrap.cc:492] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b: No bootstrap required, opened a new log
I20260925 02:31:22.454859 11353 ts_tablet_manager.cc:1397] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:31:22.456611 11354 tablet_bootstrap.cc:492] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0: Bootstrap starting.
I20260925 02:31:22.457520 11354 tablet_bootstrap.cc:654] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:22.457466 11353 raft_consensus.cc:348] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.457692 11353 raft_consensus.cc:374] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:22.457733 11353 raft_consensus.cc:729] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e4dcff793f40e7b95622082036bd2b, State: Initialized, Role: FOLLOWER
I20260925 02:31:22.457947 11353 consensus_queue.cc:260] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.458292 11353 ts_tablet_manager.cc:1428] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 02:31:22.458336 11336 heartbeater.cc:503] Master 127.10.13.62:44219 was elected leader, sending a full tablet report...
I20260925 02:31:22.458418 11354 log.cc:824] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:22.458804 11357 tablet_bootstrap.cc:492] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01: Bootstrap starting.
I20260925 02:31:22.459321 11354 tablet_bootstrap.cc:492] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0: No bootstrap required, opened a new log
I20260925 02:31:22.459478 11354 ts_tablet_manager.cc:1397] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:22.459750 11357 tablet_bootstrap.cc:654] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:22.460588 11357 log.cc:824] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:22.461481 11357 tablet_bootstrap.cc:492] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01: No bootstrap required, opened a new log
I20260925 02:31:22.461598 11357 ts_tablet_manager.cc:1397] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:31:22.462154 11354 raft_consensus.cc:348] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.462312 11354 raft_consensus.cc:374] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:22.462335 11354 raft_consensus.cc:729] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d856ffd0164cbbac3f26b28cb0bfa0, State: Initialized, Role: FOLLOWER
I20260925 02:31:22.462504 11354 consensus_queue.cc:260] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.462888 11201 heartbeater.cc:503] Master 127.10.13.62:44219 was elected leader, sending a full tablet report...
I20260925 02:31:22.462965 11354 ts_tablet_manager.cc:1428] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:22.464178 11357 raft_consensus.cc:348] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.464321 11357 raft_consensus.cc:374] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:22.464344 11357 raft_consensus.cc:729] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6759a4f234443ea06ea3666d718d01, State: Initialized, Role: FOLLOWER
I20260925 02:31:22.464453 11357 consensus_queue.cc:260] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.464828 11357 ts_tablet_manager.cc:1428] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:22.465029 11066 heartbeater.cc:503] Master 127.10.13.62:44219 was elected leader, sending a full tablet report...
W20260925 02:31:22.500470 11202 tablet.cc:2369] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:31:22.586167 11067 tablet.cc:2369] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:22.616088 11362 raft_consensus.cc:482] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:22.616232 11362 raft_consensus.cc:504] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.616745 11362 leader_election.cc:288] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67d856ffd0164cbbac3f26b28cb0bfa0 (127.10.13.2:40761), 49e4dcff793f40e7b95622082036bd2b (127.10.13.3:35045)
I20260925 02:31:22.620110 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.617294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:51729 (local address 127.10.13.2:40761)
0925 02:31:22.617357 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.617359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.617366 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:31:22.617410 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.617411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.617425 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.617479 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:22.617806 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.618277 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.619298 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.619690 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.619746 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.619756 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.619939 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.619979 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:31:22.620193 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.616978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:35045 (local address 127.10.13.1:35267)
0925 02:31:22.617201 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:22.617210 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:22.617268 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:22.617432 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:22.617433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:22.617435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:22.617518 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.617520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.618307 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.618308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.619196 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.619199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.619886 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.619887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.619991 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.619996 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:22.620056 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:31:22.620094 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":65,"spinlock_wait_cycles":16768,"thread_start_us":112,"threads_started":1}
I20260925 02:31:22.620229 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.617129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:35267 (local address 127.10.13.3:35045)
0925 02:31:22.617197 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.617199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.617290 (+ 91us) server_negotiation.cc:407] Connection header received
0925 02:31:22.617351 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.617352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.617361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.617407 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:22.617717 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.618223 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.619300 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.619803 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.619875 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.619887 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.620103 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.620154 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:31:22.620290 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.616978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:40761 (local address 127.10.13.1:51729)
0925 02:31:22.617153 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:22.617166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:22.617242 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:22.617512 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:22.617513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:22.617516 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:22.617722 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.617725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.618360 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.618361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.619199 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:22.619201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.619777 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.619779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:22.619857 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.619863 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:31:22.619919 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:31:22.619964 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":141,"threads_started":1}
I20260925 02:31:22.620682 11290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da93521f6b6b4ae2926cff09e885dcb3" candidate_uuid: "ff6759a4f234443ea06ea3666d718d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e4dcff793f40e7b95622082036bd2b" is_pre_election: true
I20260925 02:31:22.620735 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da93521f6b6b4ae2926cff09e885dcb3" candidate_uuid: "ff6759a4f234443ea06ea3666d718d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" is_pre_election: true
I20260925 02:31:22.620864 11290 raft_consensus.cc:2454] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6759a4f234443ea06ea3666d718d01 in term 0.
I20260925 02:31:22.620934 11154 raft_consensus.cc:2454] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6759a4f234443ea06ea3666d718d01 in term 0.
I20260925 02:31:22.621151 10952 leader_election.cc:302] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e4dcff793f40e7b95622082036bd2b, ff6759a4f234443ea06ea3666d718d01; no voters:
I20260925 02:31:22.621294 11362 raft_consensus.cc:2781] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:22.621364 11362 raft_consensus.cc:482] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:22.621384 11362 raft_consensus.cc:3037] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:22.622327 11362 raft_consensus.cc:504] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.622512 11362 leader_election.cc:288] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [CANDIDATE]: Term 1 election: Requested vote from peers 67d856ffd0164cbbac3f26b28cb0bfa0 (127.10.13.2:40761), 49e4dcff793f40e7b95622082036bd2b (127.10.13.3:35045)
I20260925 02:31:22.622680 11290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da93521f6b6b4ae2926cff09e885dcb3" candidate_uuid: "ff6759a4f234443ea06ea3666d718d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e4dcff793f40e7b95622082036bd2b"
I20260925 02:31:22.622712 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da93521f6b6b4ae2926cff09e885dcb3" candidate_uuid: "ff6759a4f234443ea06ea3666d718d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0"
I20260925 02:31:22.622766 11290 raft_consensus.cc:3037] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:22.622777 11154 raft_consensus.cc:3037] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:22.623821 11154 raft_consensus.cc:2454] T da93521f6b6b4ae2926cff09e885dcb3 P 67d856ffd0164cbbac3f26b28cb0bfa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6759a4f234443ea06ea3666d718d01 in term 1.
I20260925 02:31:22.624028 10954 leader_election.cc:302] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d856ffd0164cbbac3f26b28cb0bfa0, ff6759a4f234443ea06ea3666d718d01; no voters:
I20260925 02:31:22.624053 11290 raft_consensus.cc:2454] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6759a4f234443ea06ea3666d718d01 in term 1.
I20260925 02:31:22.624130 11362 raft_consensus.cc:2781] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:22.624310 11362 raft_consensus.cc:686] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [term 1 LEADER]: Becoming Leader. State: Replica: ff6759a4f234443ea06ea3666d718d01, State: Running, Role: LEADER
I20260925 02:31:22.624439 11362 consensus_queue.cc:237] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } }
I20260925 02:31:22.625406 10872 catalog_manager.cc:5495] T da93521f6b6b4ae2926cff09e885dcb3 P ff6759a4f234443ea06ea3666d718d01 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff6759a4f234443ea06ea3666d718d01 (127.10.13.1). New cstate: current_term: 1 leader_uuid: "ff6759a4f234443ea06ea3666d718d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d856ffd0164cbbac3f26b28cb0bfa0" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff6759a4f234443ea06ea3666d718d01" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e4dcff793f40e7b95622082036bd2b" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 35045 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:22.635869 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.627204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.10.13.1:37007)
0925 02:31:22.627290 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.627292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.627408 (+ 116us) server_negotiation.cc:407] Connection header received
0925 02:31:22.627494 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.627495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.627508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.627564 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:22.628266 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.628944 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.629917 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.630191 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.630284 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.632897 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:22.632910 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:22.632910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:22.632919 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:22.634529 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.634961 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.634966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.634966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.635036 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.635249 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.635250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.635250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.635412 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:22.635602 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.635643 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.635731 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:22.661038 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.653046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.10.13.1:37007)
0925 02:31:22.653115 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.653117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.653272 (+ 155us) server_negotiation.cc:407] Connection header received
0925 02:31:22.653354 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.653355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.653368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.653458 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:22.654056 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.654542 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.655283 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.655486 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.655557 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.658252 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:22.658254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:22.658254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:22.658263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:22.659653 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.660057 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.660059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.660059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.660123 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.660345 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.660346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.660346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.660479 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:22.660573 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.660772 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.660844 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 02:31:22.664542 11337 tablet.cc:2369] T da93521f6b6b4ae2926cff09e885dcb3 P 49e4dcff793f40e7b95622082036bd2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[10826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:22.685660 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:22.677576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.10.13.1:37007)
0925 02:31:22.677656 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:22.677659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:22.677764 (+ 105us) server_negotiation.cc:407] Connection header received
0925 02:31:22.677879 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:22.677880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:22.677899 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:22.677974 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:22.678666 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.679193 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.680039 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:22.680251 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:22.680333 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:22.682955 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:22.682956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:22.682957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:22.682967 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:22.684375 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.684759 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.684760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.684761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.684827 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:22.685083 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:22.685084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:22.685085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:22.685220 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:22.685337 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:22.685436 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:31:22.685524 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 02:31:22.686281 10978 tablet_service.cc:2908] Rejecting scan request for tablet da93521f6b6b4ae2926cff09e885dcb3: Uninitialized: clean time has not yet been initialized
I20260925 02:31:22.687299 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 10935
I20260925 02:31:22.693714 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 11071
I20260925 02:31:22.698616 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 11206
I20260925 02:31:22.703483 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 10840
2026-09-25T02:31:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2104 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[11379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[11379](info): set up 2 sockets
Sep 25 02:31:22 dist-test-slave-q8mg krb5kdc[11379](info): commencing operation
krb5kdc: starting...
W20260925 02:31:23.755168 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:23 dist-test-slave-q8mg krb5kdc[11379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:23.879549 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40435
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:40435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:23.962100 11392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:23.962323 11392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:23.962347 11392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:23.963577 11392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:23.963626 11392 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:31:23.963640 11392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:23.963653 11392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:23.963665 11392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:23.963677 11392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:23.965322 11392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:40435
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:23.965601 11392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:23.965883 11392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:23.971091 11398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:23.971232 11397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:23.971179 11392 server_base.cc:1034] running on GCE node
W20260925 02:31:23.971124 11400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:23.971594 11392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:23.971875 11392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:23.973075 11392 hybrid_clock.cc:648] HybridClock initialized: now 1790303483973003 us; error 45 us; skew 500 ppm
Sep 25 02:31:23 dist-test-slave-q8mg krb5kdc[11379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:23.981338 11392 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:23.981838 11392 webserver.cc:460] Webserver started at http://127.10.13.62:45447/ using document root <none> and password file <none>
I20260925 02:31:23.982115 11392 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:23.982188 11392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:23.982340 11392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:23.983435 11392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "229b23787ea840fcbd2a85f77dfaed8b"
format_stamp: "Formatted at 2026-09-25 02:31:23 on dist-test-slave-q8mg"
server_key: "a2514eca6bc7932332639fe8f007313d"
server_key_iv: "0df46addd4fc07d5aac6b691e8b7d7b9"
server_key_version: "encryptionkey@0"
I20260925 02:31:23.983809 11392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "229b23787ea840fcbd2a85f77dfaed8b"
format_stamp: "Formatted at 2026-09-25 02:31:23 on dist-test-slave-q8mg"
server_key: "a2514eca6bc7932332639fe8f007313d"
server_key_iv: "0df46addd4fc07d5aac6b691e8b7d7b9"
server_key_version: "encryptionkey@0"
I20260925 02:31:23.985287 11392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:31:23.986157 11407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:23.986362 11392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:23.986476 11392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "229b23787ea840fcbd2a85f77dfaed8b"
format_stamp: "Formatted at 2026-09-25 02:31:23 on dist-test-slave-q8mg"
server_key: "a2514eca6bc7932332639fe8f007313d"
server_key_iv: "0df46addd4fc07d5aac6b691e8b7d7b9"
server_key_version: "encryptionkey@0"
I20260925 02:31:23.986558 11392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:24.000447 11392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:24.002746 11392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:24.002918 11392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:24.007850 11392 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:40435
I20260925 02:31:24.007910 11469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:40435 every 8 connection(s)
I20260925 02:31:24.008244 11392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:24.009001 11470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:24.011201 11470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b: Bootstrap starting.
I20260925 02:31:24.011749 11470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:24.011977 11470 log.cc:824] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:24.012650 11470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b: No bootstrap required, opened a new log
I20260925 02:31:24.014811 11470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b23787ea840fcbd2a85f77dfaed8b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40435 } }
I20260925 02:31:24.014952 11470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:24.014976 11470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229b23787ea840fcbd2a85f77dfaed8b, State: Initialized, Role: FOLLOWER
I20260925 02:31:24.015012 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 11392
I20260925 02:31:24.015123 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:24.015168 11470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b23787ea840fcbd2a85f77dfaed8b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40435 } }
I20260925 02:31:24.015262 11470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:24.015317 11470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:24.015326 10292 external_mini_cluster.cc:1428] Setting key 887b64e041edb9091849b5c2da2d1b17
I20260925 02:31:24.015370 11470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:24.016132 11470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b23787ea840fcbd2a85f77dfaed8b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40435 } }
I20260925 02:31:24.016243 11470 leader_election.cc:302] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 229b23787ea840fcbd2a85f77dfaed8b; no voters:
I20260925 02:31:24.016389 11470 leader_election.cc:288] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:24.016487 11475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:24.016661 11475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [term 1 LEADER]: Becoming Leader. State: Replica: 229b23787ea840fcbd2a85f77dfaed8b, State: Running, Role: LEADER
I20260925 02:31:24.016729 11470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:24.016826 11475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b23787ea840fcbd2a85f77dfaed8b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40435 } }
I20260925 02:31:24.017351 11476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "229b23787ea840fcbd2a85f77dfaed8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b23787ea840fcbd2a85f77dfaed8b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40435 } } }
I20260925 02:31:24.017452 11476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:24.017632 11477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 229b23787ea840fcbd2a85f77dfaed8b. Latest consensus state: current_term: 1 leader_uuid: "229b23787ea840fcbd2a85f77dfaed8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229b23787ea840fcbd2a85f77dfaed8b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40435 } } }
I20260925 02:31:24.017711 11477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:24.017903 11484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:24.018433 11484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:24.020058 11484 catalog_manager.cc:1347] Generated new cluster ID: 73fbaf0be3bc49c2b1a23a48da218b4b
I20260925 02:31:24.020112 11484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:24.028098 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.015834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.10.13.62:40435)
0925 02:31:24.016064 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.016075 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.016111 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:31:24.016231 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.016238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.016297 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.016452 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:24.017352 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.018867 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.019906 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.020420 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.020547 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.022955 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:24.022959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:24.022960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:24.022973 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:24.026401 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.026844 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.026848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.026848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.026963 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.027194 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.027196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.027196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.027461 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:24.027652 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.027735 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.027892 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260925 02:31:24.039644 11484 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:24.040231 11484 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:24.047694 11484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b: Generated new TSK 0
I20260925 02:31:24.047946 11484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:24.079463 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40435
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:24.161600 11496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:24.161800 11496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:24.161824 11496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:24.162937 11496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:24.162973 11496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:24.162993 11496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:24.163010 11496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:24.164439 11496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:24.164625 11496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:24.164809 11496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:24.169921 11502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:24.170029 11504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:24.170097 11501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:24.170043 11496 server_base.cc:1034] running on GCE node
I20260925 02:31:24.170460 11496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:24.170733 11496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:24.171909 11496 hybrid_clock.cc:648] HybridClock initialized: now 1790303484171881 us; error 40 us; skew 500 ppm
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:24.174769 11496 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:24.175241 11496 webserver.cc:460] Webserver started at http://127.10.13.1:36063/ using document root <none> and password file <none>
I20260925 02:31:24.175509 11496 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:24.175583 11496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:24.175733 11496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:24.176800 11496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "762c5149257e437ba541a9edbd1a7cbb"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "8d09352f492cbb7a6634f879f6215b2c"
server_key_iv: "d7e58bb16f5e90a770a29a5cc38cab0a"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.177240 11496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "762c5149257e437ba541a9edbd1a7cbb"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "8d09352f492cbb7a6634f879f6215b2c"
server_key_iv: "d7e58bb16f5e90a770a29a5cc38cab0a"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.178635 11496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:24.179411 11511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.179596 11496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:24.179706 11496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "762c5149257e437ba541a9edbd1a7cbb"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "8d09352f492cbb7a6634f879f6215b2c"
server_key_iv: "d7e58bb16f5e90a770a29a5cc38cab0a"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.179782 11496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:24.205998 11496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:24.208539 11496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:24.208725 11496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:24.209313 11496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:24.209388 11496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.209434 11496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:24.209504 11496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:24.217163 11496 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:41289
I20260925 02:31:24.217559 11627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:41289 every 8 connection(s)
I20260925 02:31:24.217624 11496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:24.219787 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.209968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:37605 (local address 127.10.13.62:40435)
0925 02:31:24.210051 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.210054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.210236 (+ 182us) server_negotiation.cc:407] Connection header received
0925 02:31:24.210749 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.210750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.210770 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.210852 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:24.211775 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.212646 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.213804 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.214190 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.214278 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.216565 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:24.216567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:24.216567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:24.216576 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:24.218436 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.218877 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.218878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.218878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.218939 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.219263 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.219263 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.219264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.219375 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:24.219432 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.219651 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.219724 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:31:24.219955 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.209917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40435 (local address 127.10.13.1:37605)
0925 02:31:24.210095 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.210139 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.210642 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.210938 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.210942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.211339 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:24.211672 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.211677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.212756 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.212757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.213693 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.213696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.214299 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.214301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.214437 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.215059 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:24.215070 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:24.216477 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:24.218553 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.218560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.218562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.218787 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.219076 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.219077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.219077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.219192 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.219424 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:24.219427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:24.219540 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.219618 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.219742 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260925 02:31:24.220466 11628 heartbeater.cc:347] Connected to a master server at 127.10.13.62:40435
I20260925 02:31:24.220556 11628 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:24.220769 11628 heartbeater.cc:511] Master 127.10.13.62:40435 requested a full tablet report, sending...
I20260925 02:31:24.221407 11423 ts_manager.cc:194] Registered new tserver with Master: 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289)
I20260925 02:31:24.222170 11423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:37605
I20260925 02:31:24.225065 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 11496
I20260925 02:31:24.225165 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:24.225363 10292 external_mini_cluster.cc:1428] Setting key a7231f05630691504c1ed253dc0b7106
I20260925 02:31:24.227378 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.221766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:50441 (local address 127.10.13.62:40435)
0925 02:31:24.221829 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.221831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.221839 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:24.221936 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.221937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.221947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.222004 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:24.222363 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.222910 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.223615 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.223820 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.223893 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.224757 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:24.224759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:24.224760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:24.224766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:24.226277 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.226686 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.226687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.226687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.226736 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.226937 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.226938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.226938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.227031 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:24.227086 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.227261 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.227320 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:31:24.227504 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.221590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40435 (local address 127.10.13.1:50441)
0925 02:31:24.221751 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.221764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.221870 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.222009 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.222010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.222204 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:24.222301 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.222303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.223012 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.223012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.223530 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.223531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.223904 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.223905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.223977 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.224373 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:24.224379 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:24.224669 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:24.226356 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.226357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.226357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.226610 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.226800 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.226801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.226802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.226875 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.227098 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:24.227099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:24.227162 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.227234 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.227352 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:24.253324 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40435
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:24.337427 11637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:24.337639 11637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:24.337663 11637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:24.338867 11637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:24.338909 11637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:24.338930 11637 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:24.338948 11637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:24.340402 11637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:24.340608 11637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:24.340801 11637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:24.346001 11642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:24.346009 11646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:24.346009 11643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:24.346314 11637 server_base.cc:1034] running on GCE node
I20260925 02:31:24.346590 11637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:24.346851 11637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:24.348018 11637 hybrid_clock.cc:648] HybridClock initialized: now 1790303484348005 us; error 47 us; skew 500 ppm
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:24.350623 11637 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:24.351099 11637 webserver.cc:460] Webserver started at http://127.10.13.2:45227/ using document root <none> and password file <none>
I20260925 02:31:24.351321 11637 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:24.351362 11637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:24.351454 11637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:24.352511 11637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "9fbc6873d14f4e03b1fa9e455798a8cc"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "05698a24a5ae8ea8f2859f82b5919db5"
server_key_iv: "2498c75802aed9078634f874dcfe2a20"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.352811 11637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "9fbc6873d14f4e03b1fa9e455798a8cc"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "05698a24a5ae8ea8f2859f82b5919db5"
server_key_iv: "2498c75802aed9078634f874dcfe2a20"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.354269 11637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:24.355088 11653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.355394 11637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:24.355484 11637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "9fbc6873d14f4e03b1fa9e455798a8cc"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "05698a24a5ae8ea8f2859f82b5919db5"
server_key_iv: "2498c75802aed9078634f874dcfe2a20"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.355549 11637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:24.377362 11637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:24.379630 11637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:24.379755 11637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:24.380262 11637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:24.380304 11637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.380330 11637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:24.380343 11637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:24.388080 11637 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:36837
I20260925 02:31:24.388168 11768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:36837 every 8 connection(s)
I20260925 02:31:24.388477 11637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:24.391927 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.380891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:35039 (local address 127.10.13.62:40435)
0925 02:31:24.380953 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.380956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.381243 (+ 287us) server_negotiation.cc:407] Connection header received
0925 02:31:24.381792 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.381794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.381809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.381895 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:24.382824 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.383632 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.385270 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.385623 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.385717 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.388596 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:24.388598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:24.388599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:24.388608 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:24.390383 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.390897 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.390899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.390899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.390970 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.391243 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.391244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.391244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.391369 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:24.391443 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.391746 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.391833 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:31:24.392133 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.380841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40435 (local address 127.10.13.2:35039)
0925 02:31:24.381101 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.381145 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.381658 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.381955 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.381959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.382414 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:24.382719 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.382726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.383738 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.383739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.385169 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.385172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.385741 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.385742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.385899 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.386489 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:24.386499 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:24.388453 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:24.390546 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.390554 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.390557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.390809 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.391041 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.391042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.391043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.391166 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.391444 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:24.391452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:24.391599 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.391720 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.391890 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260925 02:31:24.392647 11769 heartbeater.cc:347] Connected to a master server at 127.10.13.62:40435
I20260925 02:31:24.392727 11769 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:24.392974 11769 heartbeater.cc:511] Master 127.10.13.62:40435 requested a full tablet report, sending...
I20260925 02:31:24.393435 11423 ts_manager.cc:194] Registered new tserver with Master: 9fbc6873d14f4e03b1fa9e455798a8cc (127.10.13.2:36837)
I20260925 02:31:24.393823 11423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:35039
I20260925 02:31:24.398720 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 11637
I20260925 02:31:24.398828 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:24.399012 10292 external_mini_cluster.cc:1428] Setting key 2f43a00e8f84a482d8afb5a89fbbb79f
I20260925 02:31:24.399688 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.393908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:41237 (local address 127.10.13.62:40435)
0925 02:31:24.393965 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.393968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.394091 (+ 123us) server_negotiation.cc:407] Connection header received
0925 02:31:24.394158 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.394158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.394166 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.394254 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:24.394579 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.395092 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.395772 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.395952 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.396081 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.397039 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:24.397040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:24.397040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:24.397047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:24.398230 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.398585 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.398586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.398587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.398652 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.398918 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.398920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.398920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.399212 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:24.399302 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.399505 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.399593 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:31:24.399704 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.393856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40435 (local address 127.10.13.2:41237)
0925 02:31:24.394028 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.394040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.394099 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.394277 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.394277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.394436 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:24.394502 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.394504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.395187 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.395188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.395689 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.395691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.396101 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.396102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.396159 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.396575 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:24.396579 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:24.396913 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:24.398303 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.398304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.398304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.398465 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.398739 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.398740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.398740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.398813 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.399313 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:24.399314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:24.399367 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.399471 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.399594 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:24.428115 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40435
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:24.512629 11778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:24.512830 11778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:24.512856 11778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:24.514075 11778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:24.514122 11778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:24.514142 11778 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:24.514189 11778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:24.515652 11778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:24.515913 11778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:24.516208 11778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:24.521535 11787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:24.521508 11785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:24.521657 11784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:24.522013 11778 server_base.cc:1034] running on GCE node
I20260925 02:31:24.522224 11778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:24.522429 11778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:24.523569 11778 hybrid_clock.cc:648] HybridClock initialized: now 1790303484523559 us; error 27 us; skew 500 ppm
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:24.526079 11778 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:24.526505 11778 webserver.cc:460] Webserver started at http://127.10.13.3:45349/ using document root <none> and password file <none>
I20260925 02:31:24.526746 11778 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:24.526791 11778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:24.526942 11778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:24.527981 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "4fe0920b1a16434bacce34f7268e20cd"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "82dcc67e6dfb65118f12e576950c797c"
server_key_iv: "8f6d0dcf13500f62c2d54cc24f125102"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.528373 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "4fe0920b1a16434bacce34f7268e20cd"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "82dcc67e6dfb65118f12e576950c797c"
server_key_iv: "8f6d0dcf13500f62c2d54cc24f125102"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.529795 11778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:31:24.530545 11794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.530714 11778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.530822 11778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "4fe0920b1a16434bacce34f7268e20cd"
format_stamp: "Formatted at 2026-09-25 02:31:24 on dist-test-slave-q8mg"
server_key: "82dcc67e6dfb65118f12e576950c797c"
server_key_iv: "8f6d0dcf13500f62c2d54cc24f125102"
server_key_version: "encryptionkey@0"
I20260925 02:31:24.530898 11778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:24.549067 11778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:24.551110 11778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:24.551283 11778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:24.551823 11778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:24.551890 11778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.551937 11778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:24.551962 11778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.559366 11778 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:39491
I20260925 02:31:24.559406 11909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:39491 every 8 connection(s)
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:24.559808 11778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:24.562899 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.552536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:43623 (local address 127.10.13.62:40435)
0925 02:31:24.552598 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.552600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.552826 (+ 226us) server_negotiation.cc:407] Connection header received
0925 02:31:24.553521 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.553522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.553536 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.553597 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:24.554495 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.555359 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.557118 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.557428 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.557547 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.560382 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:24.560384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:24.560384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:24.560391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:24.561685 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.562080 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.562081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.562082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.562145 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.562354 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.562354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.562355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.562468 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:24.562529 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.562762 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.562834 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:31:24.563082 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.552510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40435 (local address 127.10.13.3:43623)
0925 02:31:24.552674 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.552712 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.553389 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.553671 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.553676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.554069 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:24.554372 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.554379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.555497 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.555499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.557015 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.557019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.557552 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.557553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.557671 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.558299 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:24.558322 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:24.560295 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:24.561758 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.561765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.561766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.561981 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.562207 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.562207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.562208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.562299 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.562527 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:24.562532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:24.562649 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.562732 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.562852 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260925 02:31:24.563517 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 11778
I20260925 02:31:24.563608 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:24.563640 11910 heartbeater.cc:347] Connected to a master server at 127.10.13.62:40435
I20260925 02:31:24.563728 11910 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:24.563779 10292 external_mini_cluster.cc:1428] Setting key a8f6ec5447d14f3ba538cf5cbf265356
I20260925 02:31:24.563925 11910 heartbeater.cc:511] Master 127.10.13.62:40435 requested a full tablet report, sending...
I20260925 02:31:24.564307 11423 ts_manager.cc:194] Registered new tserver with Master: 4fe0920b1a16434bacce34f7268e20cd (127.10.13.3:39491)
I20260925 02:31:24.564543 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:31:24.564817 11423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:43623
I20260925 02:31:24.572593 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.564927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:42125 (local address 127.10.13.62:40435)
0925 02:31:24.566307 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.566308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.566316 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:24.566333 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.566334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.566342 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.566391 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:24.566845 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.567466 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.568607 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.568847 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.568951 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.569957 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:24.569958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:24.569959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:24.569966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:24.571468 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.571886 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.571887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.571888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.571937 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.572142 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.572142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.572143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.572238 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:24.572290 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.572469 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.572528 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1357}
I20260925 02:31:24.572752 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.564823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40435 (local address 127.10.13.3:42125)
0925 02:31:24.564986 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.564997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.565042 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.566422 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.566423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.566675 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:24.566757 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.566759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.567571 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.567572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.568497 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.568501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.568952 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.568954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.569036 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.569470 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:24.569476 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:24.569868 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:24.571560 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.571561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.571561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.571810 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.572001 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:24.572002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:24.572003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:24.572077 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:24.572300 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:24.572301 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:24.572366 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.572441 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.572584 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:31:24 dist-test-slave-q8mg krb5kdc[11379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:24.589881 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.582275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.10.13.62:40435)
0925 02:31:24.582337 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.582339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.582348 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:24.582374 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.582375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.582384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.582441 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:24.583018 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.583589 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.584365 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.584546 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.584613 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.587223 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:24.587225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:24.587225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:24.587233 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:24.588634 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.589097 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.589099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.589099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.589162 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:24.589326 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:24.589327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:24.589327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:24.589476 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:24.589584 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.589684 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.589754 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:31:24.591792 11423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:24.593659 11423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:24.601125 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.596799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.10.13.1:41289)
0925 02:31:24.597000 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.597004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.597014 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:24.597054 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.597056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.597073 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.597412 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:24.597469 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.598612 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.599794 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.600596 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.600682 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.600748 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.600970 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.601011 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":128,"threads_started":1}
I20260925 02:31:24.601141 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.596280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:41289 (local address 127.0.0.1:45088)
0925 02:31:24.596442 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.596485 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.596551 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.597232 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.597236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.597239 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:24.597327 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.597330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.598747 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.598748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.599703 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.599706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.600699 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.600701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.600796 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.600812 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.600930 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.601010 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":8,"thread_start_us":125,"threads_started":1}
I20260925 02:31:24.601266 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.596518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.10.13.3:39491)
0925 02:31:24.596742 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.596747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.597109 (+ 362us) server_negotiation.cc:407] Connection header received
0925 02:31:24.597187 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.597190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.597207 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.597315 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:24.597569 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.598613 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.600114 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.600707 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.600784 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.600849 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.601103 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.601154 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260925 02:31:24.601526 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.596293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:39491 (local address 127.0.0.1:42836)
0925 02:31:24.597035 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.597048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.597106 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.597405 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.597406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.597409 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:24.597463 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.597465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.598705 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.598707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.600010 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.600012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.600930 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.600932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.600992 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.600997 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.601389 (+ 392us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.601441 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":71,"thread_start_us":89,"threads_started":1}
I20260925 02:31:24.601768 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.596585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.10.13.2:36837)
0925 02:31:24.596766 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.596770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.596779 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:24.596839 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.596841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.596861 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.596953 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:24.597416 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.598260 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.599934 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.600626 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.600686 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.600747 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.601095 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.601140 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260925 02:31:24.601930 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.596272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36837 (local address 127.0.0.1:56992)
0925 02:31:24.596445 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.596487 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.596569 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.597254 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.597254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.597261 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:24.597327 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.597330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.598364 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.598366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.599838 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.599841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.600785 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.600786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.600907 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.600913 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.600963 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.601011 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":83,"thread_start_us":57,"threads_started":1}
I20260925 02:31:24.602039 11562 tablet_service.cc:1467] Processing CreateTablet for tablet 2531c1d9e69842b5b8784b6961277c55 (DEFAULT_TABLE table=test-table [id=2efca832bca740bcb3f47df3dbe9257f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:24.602474 11562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2531c1d9e69842b5b8784b6961277c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:24.602664 11703 tablet_service.cc:1467] Processing CreateTablet for tablet 2531c1d9e69842b5b8784b6961277c55 (DEFAULT_TABLE table=test-table [id=2efca832bca740bcb3f47df3dbe9257f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:24.603013 11703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2531c1d9e69842b5b8784b6961277c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:24.605746 11844 tablet_service.cc:1467] Processing CreateTablet for tablet 2531c1d9e69842b5b8784b6961277c55 (DEFAULT_TABLE table=test-table [id=2efca832bca740bcb3f47df3dbe9257f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:24.605959 11934 tablet_bootstrap.cc:492] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Bootstrap starting.
I20260925 02:31:24.606218 11844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2531c1d9e69842b5b8784b6961277c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:24.606498 11934 tablet_bootstrap.cc:654] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:24.606750 11935 tablet_bootstrap.cc:492] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Bootstrap starting.
I20260925 02:31:24.607009 11934 log.cc:824] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:24.607368 11935 tablet_bootstrap.cc:654] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:24.607677 11935 log.cc:824] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:24.608004 11934 tablet_bootstrap.cc:492] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: No bootstrap required, opened a new log
I20260925 02:31:24.608081 11934 ts_tablet_manager.cc:1397] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:31:24.608500 11935 tablet_bootstrap.cc:492] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: No bootstrap required, opened a new log
I20260925 02:31:24.608568 11935 ts_tablet_manager.cc:1397] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:24.609506 11938 tablet_bootstrap.cc:492] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Bootstrap starting.
I20260925 02:31:24.610172 11934 raft_consensus.cc:348] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.610356 11934 raft_consensus.cc:374] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:24.610392 11934 raft_consensus.cc:729] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762c5149257e437ba541a9edbd1a7cbb, State: Initialized, Role: FOLLOWER
I20260925 02:31:24.610538 11938 tablet_bootstrap.cc:654] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:24.610546 11934 consensus_queue.cc:260] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.610527 11935 raft_consensus.cc:348] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.610690 11935 raft_consensus.cc:374] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:24.610733 11935 raft_consensus.cc:729] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbc6873d14f4e03b1fa9e455798a8cc, State: Initialized, Role: FOLLOWER
I20260925 02:31:24.610874 11934 ts_tablet_manager.cc:1428] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:24.610838 11935 consensus_queue.cc:260] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.610922 11938 log.cc:824] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:24.611040 11628 heartbeater.cc:503] Master 127.10.13.62:40435 was elected leader, sending a full tablet report...
I20260925 02:31:24.611110 11935 ts_tablet_manager.cc:1428] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:24.611147 11769 heartbeater.cc:503] Master 127.10.13.62:40435 was elected leader, sending a full tablet report...
I20260925 02:31:24.612171 11938 tablet_bootstrap.cc:492] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: No bootstrap required, opened a new log
I20260925 02:31:24.612313 11938 ts_tablet_manager.cc:1397] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:31:24.614493 11938 raft_consensus.cc:348] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.614642 11938 raft_consensus.cc:374] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:24.614666 11938 raft_consensus.cc:729] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0920b1a16434bacce34f7268e20cd, State: Initialized, Role: FOLLOWER
I20260925 02:31:24.614821 11938 consensus_queue.cc:260] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.615188 11938 ts_tablet_manager.cc:1428] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:24.615242 11910 heartbeater.cc:503] Master 127.10.13.62:40435 was elected leader, sending a full tablet report...
W20260925 02:31:24.639132 11770 tablet.cc:2369] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:31:24.718456 11629 tablet.cc:2369] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:24.801712 11940 raft_consensus.cc:482] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:24.801852 11940 raft_consensus.cc:504] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.802194 11940 leader_election.cc:288] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289), 4fe0920b1a16434bacce34f7268e20cd (127.10.13.3:39491)
I20260925 02:31:24.805323 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.802426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:39491 (local address 127.10.13.2:48819)
0925 02:31:24.802474 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.802485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.802552 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.802754 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.802756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.802759 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:24.802827 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.802830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.803550 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.803551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.804468 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.804471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.805065 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.805066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.805141 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.805145 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.805198 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.805229 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":31}
I20260925 02:31:24.805337 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.802545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:48819 (local address 127.10.13.3:39491)
0925 02:31:24.802594 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.802596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.802604 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:24.802635 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.802636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.802649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.802712 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:24.802931 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.803465 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.804616 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.804987 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.805049 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.805059 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.805219 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.805262 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:31:24.805419 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.802529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:58647 (local address 127.10.13.1:41289)
0925 02:31:24.802580 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.802583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.802756 (+ 173us) server_negotiation.cc:407] Connection header received
0925 02:31:24.802817 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.802818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.802832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.802909 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:24.803083 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.803570 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.804574 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.805034 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.805104 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.805114 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.805306 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.805348 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:31:24.805480 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.802426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:41289 (local address 127.10.13.2:58647)
0925 02:31:24.802680 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:24.802688 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:24.802748 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:24.802926 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:24.802927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:24.802929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:24.803011 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.803013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.803652 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.803653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.804471 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:24.804474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.805136 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.805137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:24.805228 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.805233 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:24.805289 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:31:24.805327 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":72,"thread_start_us":128,"threads_started":1}
I20260925 02:31:24.805768 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2531c1d9e69842b5b8784b6961277c55" candidate_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0920b1a16434bacce34f7268e20cd" is_pre_election: true
I20260925 02:31:24.805832 11582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2531c1d9e69842b5b8784b6961277c55" candidate_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762c5149257e437ba541a9edbd1a7cbb" is_pre_election: true
I20260925 02:31:24.805929 11864 raft_consensus.cc:2454] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in term 0.
I20260925 02:31:24.805997 11582 raft_consensus.cc:2454] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in term 0.
I20260925 02:31:24.806277 11655 leader_election.cc:302] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762c5149257e437ba541a9edbd1a7cbb, 9fbc6873d14f4e03b1fa9e455798a8cc; no voters:
I20260925 02:31:24.806432 11940 raft_consensus.cc:2781] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:24.806488 11940 raft_consensus.cc:482] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:24.806510 11940 raft_consensus.cc:3037] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:24.807699 11940 raft_consensus.cc:504] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.807852 11940 leader_election.cc:288] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [CANDIDATE]: Term 1 election: Requested vote from peers 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289), 4fe0920b1a16434bacce34f7268e20cd (127.10.13.3:39491)
I20260925 02:31:24.808029 11582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2531c1d9e69842b5b8784b6961277c55" candidate_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762c5149257e437ba541a9edbd1a7cbb"
I20260925 02:31:24.808130 11582 raft_consensus.cc:3037] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:24.808187 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2531c1d9e69842b5b8784b6961277c55" candidate_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0920b1a16434bacce34f7268e20cd"
I20260925 02:31:24.808288 11864 raft_consensus.cc:3037] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:24.808991 11582 raft_consensus.cc:2454] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in term 1.
I20260925 02:31:24.809157 11864 raft_consensus.cc:2454] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in term 1.
I20260925 02:31:24.809361 11654 leader_election.cc:302] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0920b1a16434bacce34f7268e20cd, 9fbc6873d14f4e03b1fa9e455798a8cc; no voters:
I20260925 02:31:24.809475 11940 raft_consensus.cc:2781] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:24.809659 11940 raft_consensus.cc:686] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 LEADER]: Becoming Leader. State: Replica: 9fbc6873d14f4e03b1fa9e455798a8cc, State: Running, Role: LEADER
I20260925 02:31:24.809779 11940 consensus_queue.cc:237] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
W20260925 02:31:24.810362 11911 tablet.cc:2369] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:24.810704 11422 catalog_manager.cc:5495] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbc6873d14f4e03b1fa9e455798a8cc (127.10.13.2). New cstate: current_term: 1 leader_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } health_report { overall_health: UNKNOWN } } }
I20260925 02:31:24.828483 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:24.825785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.10.13.2:36837)
0925 02:31:24.825850 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:24.825852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:24.825862 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:24.825941 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:24.825942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:24.825965 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:24.826059 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:24.826198 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.826680 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.827474 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:24.827686 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:24.827770 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:24.827980 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:24.828121 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:24.828217 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:24.828266 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 02:31:24.828289 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:31:24.830564 11864 raft_consensus.cc:1261] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Refusing update from remote peer 9fbc6873d14f4e03b1fa9e455798a8cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:24.830564 11582 raft_consensus.cc:1261] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Refusing update from remote peer 9fbc6873d14f4e03b1fa9e455798a8cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:24.830950 11944 consensus_queue.cc:1035] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:24.831157 11940 consensus_queue.cc:1035] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:24.834173 11948 mvcc.cc:205] Tried to move back new op lower bound from 7333083073862541312 to 7333083073781514240. Current Snapshot: MvccSnapshot[applied={T|T < 7333083073862541312}]
I20260925 02:31:24.838145 11422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:24.838253 11422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:24.839097 11422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b: Sending DeleteTablet for 3 replicas of tablet 2531c1d9e69842b5b8784b6961277c55
I20260925 02:31:24.839496 11562 tablet_service.cc:1514] Processing DeleteTablet for tablet 2531c1d9e69842b5b8784b6961277c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:24 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:45088
I20260925 02:31:24.839584 11703 tablet_service.cc:1514] Processing DeleteTablet for tablet 2531c1d9e69842b5b8784b6961277c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:24 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:56992
I20260925 02:31:24.839650 11844 tablet_service.cc:1514] Processing DeleteTablet for tablet 2531c1d9e69842b5b8784b6961277c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:24 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:42836
I20260925 02:31:24.839880 11958 tablet_replica.cc:330] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: stopping tablet replica
I20260925 02:31:24.839986 11958 raft_consensus.cc:2229] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:24.840101 11959 tablet_replica.cc:330] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: stopping tablet replica
I20260925 02:31:24.840107 11958 raft_consensus.cc:2258] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:24.840377 11957 tablet_replica.cc:330] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: stopping tablet replica
I20260925 02:31:24.840405 11959 raft_consensus.cc:2229] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:24.840359 11422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:24.840508 11422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:24.840543 11959 raft_consensus.cc:2258] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:24.840744 11957 raft_consensus.cc:2229] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:24.840956 11958 ts_tablet_manager.cc:1905] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:24.841450 11959 ts_tablet_manager.cc:1905] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:24.842651 11703 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd29cdea6494a009e7b5e580bdf0cba (DEFAULT_TABLE table=test-table [id=bf269fbb1ca0474aadd4029cfa4b290b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:24.842762 11703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd29cdea6494a009e7b5e580bdf0cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:24.843246 11959 ts_tablet_manager.cc:1918] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:24.843256 11958 ts_tablet_manager.cc:1918] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:24.843312 11959 log.cc:1196] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/2531c1d9e69842b5b8784b6961277c55
I20260925 02:31:24.843320 11958 log.cc:1196] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/2531c1d9e69842b5b8784b6961277c55
I20260925 02:31:24.843605 11958 ts_tablet_manager.cc:1939] T 2531c1d9e69842b5b8784b6961277c55 P 9fbc6873d14f4e03b1fa9e455798a8cc: Deleting consensus metadata
I20260925 02:31:24.843612 11959 ts_tablet_manager.cc:1939] T 2531c1d9e69842b5b8784b6961277c55 P 4fe0920b1a16434bacce34f7268e20cd: Deleting consensus metadata
I20260925 02:31:24.844120 11844 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd29cdea6494a009e7b5e580bdf0cba (DEFAULT_TABLE table=test-table [id=bf269fbb1ca0474aadd4029cfa4b290b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:24.844219 11844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd29cdea6494a009e7b5e580bdf0cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:24.844343 11408 catalog_manager.cc:4841] TS 4fe0920b1a16434bacce34f7268e20cd (127.10.13.3:39491): tablet 2531c1d9e69842b5b8784b6961277c55 (table test-table [id=2efca832bca740bcb3f47df3dbe9257f]) successfully deleted
I20260925 02:31:24.844990 11935 tablet_bootstrap.cc:492] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: Bootstrap starting.
I20260925 02:31:24.845466 11935 tablet_bootstrap.cc:654] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:24.845520 11938 tablet_bootstrap.cc:492] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: Bootstrap starting.
I20260925 02:31:24.846060 11938 tablet_bootstrap.cc:654] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:24.846230 11935 tablet_bootstrap.cc:492] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: No bootstrap required, opened a new log
I20260925 02:31:24.846280 11935 ts_tablet_manager.cc:1397] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:24.846401 11562 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd29cdea6494a009e7b5e580bdf0cba (DEFAULT_TABLE table=test-table [id=bf269fbb1ca0474aadd4029cfa4b290b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:24.846441 11935 raft_consensus.cc:348] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.846509 11935 raft_consensus.cc:374] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:24.846519 11562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd29cdea6494a009e7b5e580bdf0cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:24.846531 11935 raft_consensus.cc:729] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbc6873d14f4e03b1fa9e455798a8cc, State: Initialized, Role: FOLLOWER
I20260925 02:31:24.846587 11935 consensus_queue.cc:260] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.846580 11411 catalog_manager.cc:4841] TS 9fbc6873d14f4e03b1fa9e455798a8cc (127.10.13.2:36837): tablet 2531c1d9e69842b5b8784b6961277c55 (table test-table [id=2efca832bca740bcb3f47df3dbe9257f]) successfully deleted
I20260925 02:31:24.846676 11935 ts_tablet_manager.cc:1428] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:24.847973 11934 tablet_bootstrap.cc:492] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: Bootstrap starting.
I20260925 02:31:24.848104 11957 raft_consensus.cc:2258] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:24.848379 11934 tablet_bootstrap.cc:654] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:24.848527 11938 tablet_bootstrap.cc:492] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: No bootstrap required, opened a new log
I20260925 02:31:24.848599 11938 ts_tablet_manager.cc:1397] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:31:24.848762 11938 raft_consensus.cc:348] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.848840 11938 raft_consensus.cc:374] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:24.848866 11938 raft_consensus.cc:729] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0920b1a16434bacce34f7268e20cd, State: Initialized, Role: FOLLOWER
I20260925 02:31:24.848928 11938 consensus_queue.cc:260] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.849042 11938 ts_tablet_manager.cc:1428] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:24.849453 11957 ts_tablet_manager.cc:1905] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:24.849601 11934 tablet_bootstrap.cc:492] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: No bootstrap required, opened a new log
I20260925 02:31:24.849676 11934 ts_tablet_manager.cc:1397] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:24.849938 11934 raft_consensus.cc:348] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.850019 11934 raft_consensus.cc:374] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:24.850034 11934 raft_consensus.cc:729] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762c5149257e437ba541a9edbd1a7cbb, State: Initialized, Role: FOLLOWER
I20260925 02:31:24.850109 11934 consensus_queue.cc:260] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:24.850234 11934 ts_tablet_manager.cc:1428] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:24.851393 11957 ts_tablet_manager.cc:1918] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:24.851500 11957 log.cc:1196] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/2531c1d9e69842b5b8784b6961277c55
I20260925 02:31:24.852205 11957 ts_tablet_manager.cc:1939] T 2531c1d9e69842b5b8784b6961277c55 P 762c5149257e437ba541a9edbd1a7cbb: Deleting consensus metadata
I20260925 02:31:24.852679 11409 catalog_manager.cc:4841] TS 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289): tablet 2531c1d9e69842b5b8784b6961277c55 (table test-table [id=2efca832bca740bcb3f47df3dbe9257f]) successfully deleted
I20260925 02:31:25.111925 11942 raft_consensus.cc:482] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:25.112033 11942 raft_consensus.cc:504] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.112385 11942 leader_election.cc:288] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289), 9fbc6873d14f4e03b1fa9e455798a8cc (127.10.13.2:36837)
I20260925 02:31:25.112746 11944 raft_consensus.cc:482] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:25.112844 11944 raft_consensus.cc:504] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.113042 11944 leader_election.cc:288] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289), 4fe0920b1a16434bacce34f7268e20cd (127.10.13.3:39491)
I20260925 02:31:25.113227 11582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29cdea6494a009e7b5e580bdf0cba" candidate_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762c5149257e437ba541a9edbd1a7cbb" is_pre_election: true
I20260925 02:31:25.113338 11582 raft_consensus.cc:2454] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in term 0.
I20260925 02:31:25.113433 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29cdea6494a009e7b5e580bdf0cba" candidate_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0920b1a16434bacce34f7268e20cd" is_pre_election: true
I20260925 02:31:25.113510 11864 raft_consensus.cc:2454] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in term 0.
I20260925 02:31:25.113672 11655 leader_election.cc:302] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762c5149257e437ba541a9edbd1a7cbb, 9fbc6873d14f4e03b1fa9e455798a8cc; no voters:
I20260925 02:31:25.113853 11944 raft_consensus.cc:2781] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:25.113924 11944 raft_consensus.cc:482] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:25.113962 11944 raft_consensus.cc:3037] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:25.114976 11944 raft_consensus.cc:504] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.115111 11944 leader_election.cc:288] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [CANDIDATE]: Term 1 election: Requested vote from peers 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289), 4fe0920b1a16434bacce34f7268e20cd (127.10.13.3:39491)
I20260925 02:31:25.115288 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29cdea6494a009e7b5e580bdf0cba" candidate_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0920b1a16434bacce34f7268e20cd"
I20260925 02:31:25.115350 11864 raft_consensus.cc:3037] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:25.115345 11582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29cdea6494a009e7b5e580bdf0cba" candidate_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762c5149257e437ba541a9edbd1a7cbb"
I20260925 02:31:25.115422 11582 raft_consensus.cc:3037] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:25.116022 11864 raft_consensus.cc:2454] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in term 1.
I20260925 02:31:25.116204 11654 leader_election.cc:302] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0920b1a16434bacce34f7268e20cd, 9fbc6873d14f4e03b1fa9e455798a8cc; no voters:
I20260925 02:31:25.116472 11582 raft_consensus.cc:2454] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in term 1.
I20260925 02:31:25.116520 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.112827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:39659 (local address 127.10.13.2:36837)
0925 02:31:25.112895 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:25.112897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:25.113020 (+ 123us) server_negotiation.cc:407] Connection header received
0925 02:31:25.113135 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:25.113137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:25.113147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:25.113178 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:25.113407 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.114195 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.115575 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.116100 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.116203 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.116218 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:25.116414 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 02:31:25.116459 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:31:25.116549 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.112592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36837 (local address 127.10.13.3:39659)
0925 02:31:25.112949 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:25.112961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:25.113050 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:25.113250 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:25.113251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:25.113253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:25.113324 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.113326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.114291 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.114292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.115487 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.115490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.116255 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.116256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.116329 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.116334 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:25.116395 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 02:31:25.116450 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260925 02:31:25.116930 11944 raft_consensus.cc:2781] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:25.116930 11723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29cdea6494a009e7b5e580bdf0cba" candidate_uuid: "4fe0920b1a16434bacce34f7268e20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" is_pre_election: true
I20260925 02:31:25.117071 11944 raft_consensus.cc:686] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 LEADER]: Becoming Leader. State: Replica: 9fbc6873d14f4e03b1fa9e455798a8cc, State: Running, Role: LEADER
I20260925 02:31:25.117117 11944 consensus_queue.cc:237] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.117250 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.112816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:44957 (local address 127.10.13.1:41289)
0925 02:31:25.112894 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:25.112897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:25.113120 (+ 223us) server_negotiation.cc:407] Connection header received
0925 02:31:25.113232 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:25.113233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:25.113243 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:25.113272 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:25.113504 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.114208 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.115302 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.116620 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.116710 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.116719 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:25.117136 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 02:31:25.117183 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:31:25.117458 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.112742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:41289 (local address 127.10.13.3:44957)
0925 02:31:25.113047 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:25.113060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:25.113154 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:25.113337 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:25.113338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:25.113340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:25.113411 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.113413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.114316 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.114317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.115200 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.115203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.116962 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.116963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.117036 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.117040 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:25.117326 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 02:31:25.117376 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":28,"thread_start_us":61,"threads_started":1}
I20260925 02:31:25.117712 11582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29cdea6494a009e7b5e580bdf0cba" candidate_uuid: "4fe0920b1a16434bacce34f7268e20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762c5149257e437ba541a9edbd1a7cbb" is_pre_election: true
I20260925 02:31:25.117802 11582 raft_consensus.cc:2379] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fe0920b1a16434bacce34f7268e20cd in current term 1: Already voted for candidate 9fbc6873d14f4e03b1fa9e455798a8cc in this term.
I20260925 02:31:25.117839 11422 catalog_manager.cc:5495] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbc6873d14f4e03b1fa9e455798a8cc (127.10.13.2). New cstate: current_term: 1 leader_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } health_report { overall_health: UNKNOWN } } }
I20260925 02:31:25.117974 11796 leader_election.cc:302] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fe0920b1a16434bacce34f7268e20cd; no voters: 762c5149257e437ba541a9edbd1a7cbb, 9fbc6873d14f4e03b1fa9e455798a8cc
I20260925 02:31:25.118089 11942 raft_consensus.cc:2726] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 25 02:31:25 dist-test-slave-q8mg krb5kdc[11379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:25.152910 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.145699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40435 (local address 127.0.0.1:34492)
0925 02:31:25.145918 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:25.145929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:25.146005 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:25.146202 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:25.146203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:25.146537 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:25.146597 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.146600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.147272 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.147273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.147810 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.147812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.148178 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.148179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.148224 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.148732 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:25.148737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:25.150300 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:25.151782 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:25.151789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:25.151792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:25.151996 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:25.152217 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:25.152218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:25.152218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:25.152295 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:25.152576 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:25.152583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:25.152631 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 02:31:25.152704 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:31:25.152809 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":108,"threads_started":1}
I20260925 02:31:25.153128 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.145757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.10.13.62:40435)
0925 02:31:25.145937 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:25.145939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:25.146004 (+ 65us) server_negotiation.cc:407] Connection header received
0925 02:31:25.146056 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:25.146056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:25.146070 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:25.146145 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:25.146682 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.147195 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.147890 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.148104 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.148160 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.150388 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:25.150390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:25.150390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:25.150396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:25.151711 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:25.152061 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:25.152062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:25.152063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:25.152114 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:25.152363 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:25.152364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:25.152364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:25.152469 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:25.152528 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:25.152737 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 02:31:25.152859 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":154,"thread_start_us":119,"threads_started":1}
I20260925 02:31:25.155192 11422 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:34492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:31:25.155391 11422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:25.156903 11703 tablet_service.cc:1467] Processing CreateTablet for tablet c2c807a6d1bb4212a3bcc677282794fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4cbb784932cc41d28402756d8fc88b8a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:25.156913 11562 tablet_service.cc:1467] Processing CreateTablet for tablet c2c807a6d1bb4212a3bcc677282794fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4cbb784932cc41d28402756d8fc88b8a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:25.156941 11844 tablet_service.cc:1467] Processing CreateTablet for tablet c2c807a6d1bb4212a3bcc677282794fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4cbb784932cc41d28402756d8fc88b8a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:25.157080 11703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c807a6d1bb4212a3bcc677282794fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:25.157105 11562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c807a6d1bb4212a3bcc677282794fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:25.157125 11844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c807a6d1bb4212a3bcc677282794fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:25.158627 11935 tablet_bootstrap.cc:492] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: Bootstrap starting.
I20260925 02:31:25.158629 11934 tablet_bootstrap.cc:492] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: Bootstrap starting.
I20260925 02:31:25.159094 11935 tablet_bootstrap.cc:654] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:25.159308 11934 tablet_bootstrap.cc:654] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:25.159700 11935 tablet_bootstrap.cc:492] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: No bootstrap required, opened a new log
I20260925 02:31:25.159752 11935 ts_tablet_manager.cc:1397] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:25.159771 11938 tablet_bootstrap.cc:492] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: Bootstrap starting.
I20260925 02:31:25.159929 11935 raft_consensus.cc:348] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.160005 11935 raft_consensus.cc:374] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:25.160030 11935 raft_consensus.cc:729] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbc6873d14f4e03b1fa9e455798a8cc, State: Initialized, Role: FOLLOWER
I20260925 02:31:25.160089 11935 consensus_queue.cc:260] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.160198 11935 ts_tablet_manager.cc:1428] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:25.160236 11952 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.160346 11952 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:25.160409 11938 tablet_bootstrap.cc:654] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:25.160853 11934 tablet_bootstrap.cc:492] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: No bootstrap required, opened a new log
I20260925 02:31:25.160928 11934 ts_tablet_manager.cc:1397] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:25.161253 11934 raft_consensus.cc:348] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.161375 11934 raft_consensus.cc:374] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:25.161398 11934 raft_consensus.cc:729] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762c5149257e437ba541a9edbd1a7cbb, State: Initialized, Role: FOLLOWER
I20260925 02:31:25.161451 11938 tablet_bootstrap.cc:492] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: No bootstrap required, opened a new log
I20260925 02:31:25.161444 11934 consensus_queue.cc:260] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.161504 11938 ts_tablet_manager.cc:1397] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:31:25.161554 11934 ts_tablet_manager.cc:1428] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:31:25.161657 11938 raft_consensus.cc:348] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.161726 11938 raft_consensus.cc:374] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:25.161666 11939 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.161746 11938 raft_consensus.cc:729] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0920b1a16434bacce34f7268e20cd, State: Initialized, Role: FOLLOWER
I20260925 02:31:25.161764 11939 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:25.161809 11938 consensus_queue.cc:260] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.161907 11938 ts_tablet_manager.cc:1428] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:25.162029 11942 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.162124 11942 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:25.182139 11942 raft_consensus.cc:482] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:25.182255 11942 raft_consensus.cc:504] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.182461 11942 leader_election.cc:288] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289), 9fbc6873d14f4e03b1fa9e455798a8cc (127.10.13.2:36837)
I20260925 02:31:25.182657 11723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c807a6d1bb4212a3bcc677282794fb" candidate_uuid: "4fe0920b1a16434bacce34f7268e20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" is_pre_election: true
I20260925 02:31:25.182708 11582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c807a6d1bb4212a3bcc677282794fb" candidate_uuid: "4fe0920b1a16434bacce34f7268e20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762c5149257e437ba541a9edbd1a7cbb" is_pre_election: true
I20260925 02:31:25.182780 11723 raft_consensus.cc:2454] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe0920b1a16434bacce34f7268e20cd in term 0.
I20260925 02:31:25.182838 11582 raft_consensus.cc:2454] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe0920b1a16434bacce34f7268e20cd in term 0.
I20260925 02:31:25.183012 11798 leader_election.cc:302] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0920b1a16434bacce34f7268e20cd, 9fbc6873d14f4e03b1fa9e455798a8cc; no voters:
I20260925 02:31:25.183137 11942 raft_consensus.cc:2781] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:25.183182 11942 raft_consensus.cc:482] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:25.183212 11942 raft_consensus.cc:3037] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:25.184221 11942 raft_consensus.cc:504] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.184365 11942 leader_election.cc:288] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [CANDIDATE]: Term 1 election: Requested vote from peers 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289), 9fbc6873d14f4e03b1fa9e455798a8cc (127.10.13.2:36837)
I20260925 02:31:25.184508 11582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c807a6d1bb4212a3bcc677282794fb" candidate_uuid: "4fe0920b1a16434bacce34f7268e20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762c5149257e437ba541a9edbd1a7cbb"
I20260925 02:31:25.184592 11582 raft_consensus.cc:3037] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:25.184540 11723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c807a6d1bb4212a3bcc677282794fb" candidate_uuid: "4fe0920b1a16434bacce34f7268e20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc"
I20260925 02:31:25.184633 11723 raft_consensus.cc:3037] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:25.185312 11582 raft_consensus.cc:2454] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe0920b1a16434bacce34f7268e20cd in term 1.
I20260925 02:31:25.185488 11723 raft_consensus.cc:2454] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe0920b1a16434bacce34f7268e20cd in term 1.
I20260925 02:31:25.185500 11796 leader_election.cc:302] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0920b1a16434bacce34f7268e20cd, 762c5149257e437ba541a9edbd1a7cbb; no voters:
I20260925 02:31:25.185618 11942 raft_consensus.cc:2781] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:25.185752 11942 raft_consensus.cc:686] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [term 1 LEADER]: Becoming Leader. State: Replica: 4fe0920b1a16434bacce34f7268e20cd, State: Running, Role: LEADER
I20260925 02:31:25.185873 11942 consensus_queue.cc:237] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } }
I20260925 02:31:25.186386 11976 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: TxnStatusTablet state changed. Reason: New leader 4fe0920b1a16434bacce34f7268e20cd. Latest consensus state: current_term: 1 leader_uuid: "4fe0920b1a16434bacce34f7268e20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.186520 11976 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:25.186609 11422 catalog_manager.cc:5495] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe0920b1a16434bacce34f7268e20cd (127.10.13.3). New cstate: current_term: 1 leader_uuid: "4fe0920b1a16434bacce34f7268e20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } health_report { overall_health: HEALTHY } } }
I20260925 02:31:25.186846 11978 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:31:25.186983 11978 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:31:25.199417 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.196968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.10.13.3:39491)
0925 02:31:25.197031 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:25.197034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:25.197041 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:31:25.197079 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:25.197080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:25.197097 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:25.197172 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:25.197305 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.197808 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.198606 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.198807 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.198885 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.199076 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:25.199176 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:25.199261 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:25.199310 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 02:31:25.199334 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:31:25.199425 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.196776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:39491 (local address 127.0.0.1:42846)
0925 02:31:25.196900 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:25.196913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:25.196961 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:25.197180 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:25.197181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:25.197182 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:25.197237 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.197239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.197896 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.197897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.198512 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.198514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.198884 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.198885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.198943 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.198964 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:25.199246 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:25.199257 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:31:25.199307 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 02:31:25.199356 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:31:25.200680 11582 raft_consensus.cc:1261] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Refusing update from remote peer 4fe0920b1a16434bacce34f7268e20cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:25.200743 11723 raft_consensus.cc:1261] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 FOLLOWER]: Refusing update from remote peer 4fe0920b1a16434bacce34f7268e20cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:25.200975 11976 consensus_queue.cc:1035] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:25.201081 11942 consensus_queue.cc:1035] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:25.201771 11939 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: TxnStatusTablet state changed. Reason: New leader 4fe0920b1a16434bacce34f7268e20cd. Latest consensus state: current_term: 1 leader_uuid: "4fe0920b1a16434bacce34f7268e20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.201872 11939 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:25.201922 11952 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: TxnStatusTablet state changed. Reason: New leader 4fe0920b1a16434bacce34f7268e20cd. Latest consensus state: current_term: 1 leader_uuid: "4fe0920b1a16434bacce34f7268e20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.202029 11952 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:25.202446 11976 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe0920b1a16434bacce34f7268e20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.202544 11976 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:25.203408 11942 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fe0920b1a16434bacce34f7268e20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.203516 11942 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 4fe0920b1a16434bacce34f7268e20cd: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:25.203500 11949 mvcc.cc:205] Tried to move back new op lower bound from 7333083075379580928 to 7333083075322032128. Current Snapshot: MvccSnapshot[applied={T|T < 7333083075379580928}]
I20260925 02:31:25.203608 11939 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe0920b1a16434bacce34f7268e20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.203681 11939 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 762c5149257e437ba541a9edbd1a7cbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:25.204419 11952 tablet_replica.cc:439] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fe0920b1a16434bacce34f7268e20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 } } peers { permanent_uuid: "9fbc6873d14f4e03b1fa9e455798a8cc" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36837 } } peers { permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 } } }
I20260925 02:31:25.204506 11952 tablet_replica.cc:442] T c2c807a6d1bb4212a3bcc677282794fb P 9fbc6873d14f4e03b1fa9e455798a8cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:25.210177 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.206917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:39491 (local address 127.10.13.2:45797)
0925 02:31:25.207131 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:25.207140 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:25.207194 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:25.207326 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:25.207326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:25.207328 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:25.207373 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.207375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.208271 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.208273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.209145 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.209150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.209658 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.209659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.209709 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.209726 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:25.210004 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:25.210006 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:31:25.210051 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 02:31:25.210098 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260925 02:31:25.210176 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.206980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:45797 (local address 127.10.13.3:39491)
0925 02:31:25.207047 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:25.207049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:25.207204 (+ 155us) server_negotiation.cc:407] Connection header received
0925 02:31:25.207234 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:25.207235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:25.207249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:25.207325 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:25.207429 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.208171 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.209244 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.209559 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.209649 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.209844 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:25.209881 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:25.209938 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:25.210075 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 02:31:25.210097 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:31:25.212893 11582 raft_consensus.cc:1261] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Refusing update from remote peer 9fbc6873d14f4e03b1fa9e455798a8cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:25.212922 11864 raft_consensus.cc:1261] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Refusing update from remote peer 9fbc6873d14f4e03b1fa9e455798a8cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:25.213116 11952 consensus_queue.cc:1035] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe0920b1a16434bacce34f7268e20cd" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:25.213408 11952 consensus_queue.cc:1035] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "762c5149257e437ba541a9edbd1a7cbb" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:25.214747 11947 mvcc.cc:205] Tried to move back new op lower bound from 7333083075430268928 to 7333083075040038912. Current Snapshot: MvccSnapshot[applied={T|T < 7333083075430268928}]
I20260925 02:31:25.224748 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.220808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:43459 (local address 127.10.13.2:36837)
0925 02:31:25.220861 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:25.220863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:25.221115 (+ 252us) server_negotiation.cc:407] Connection header received
0925 02:31:25.221220 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:25.221221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:25.221236 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:25.221269 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:25.221623 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.222492 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.223872 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.224139 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.224242 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.224460 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:25.224499 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:25.224575 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:25.224643 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 02:31:25.224667 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 02:31:25.224982 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.220569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36837 (local address 127.10.13.3:43459)
0925 02:31:25.221040 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:25.221054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:25.221144 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:25.221345 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:25.221346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:25.221348 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:25.221522 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.221525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.222606 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.222607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.223773 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:25.223776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.224245 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.224246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:25.224325 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.224346 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:25.224575 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:25.224578 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:25.224636 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:31:25.224683 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":131,"threads_started":1}
I20260925 02:31:25.237426 11424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:25.237551 11424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:25.238287 11424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 229b23787ea840fcbd2a85f77dfaed8b: Sending DeleteTablet for 3 replicas of tablet 1dd29cdea6494a009e7b5e580bdf0cba
I20260925 02:31:25.238580 11562 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd29cdea6494a009e7b5e580bdf0cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:25 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:45088
I20260925 02:31:25.238606 11703 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd29cdea6494a009e7b5e580bdf0cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:25 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:56992
I20260925 02:31:25.238691 11957 tablet_replica.cc:330] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: stopping tablet replica
I20260925 02:31:25.238708 11958 tablet_replica.cc:330] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: stopping tablet replica
I20260925 02:31:25.238749 11957 raft_consensus.cc:2229] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:25.238762 11958 raft_consensus.cc:2229] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:25.238785 11957 raft_consensus.cc:2258] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:25.238821 11958 raft_consensus.cc:2258] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:25.239002 11844 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd29cdea6494a009e7b5e580bdf0cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:25 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:42836
I20260925 02:31:25.239086 11959 tablet_replica.cc:330] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: stopping tablet replica
I20260925 02:31:25.239130 11959 raft_consensus.cc:2229] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:25.239141 11958 ts_tablet_manager.cc:1905] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:25.239167 11959 raft_consensus.cc:2258] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:25.239419 11957 ts_tablet_manager.cc:1905] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:25.239517 11959 ts_tablet_manager.cc:1905] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:25.241511 11959 ts_tablet_manager.cc:1918] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:25.241530 11958 ts_tablet_manager.cc:1918] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:25.241600 11958 log.cc:1196] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/1dd29cdea6494a009e7b5e580bdf0cba
I20260925 02:31:25.241600 11959 log.cc:1196] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/1dd29cdea6494a009e7b5e580bdf0cba
I20260925 02:31:25.241833 11959 ts_tablet_manager.cc:1939] T 1dd29cdea6494a009e7b5e580bdf0cba P 4fe0920b1a16434bacce34f7268e20cd: Deleting consensus metadata
I20260925 02:31:25.241833 11958 ts_tablet_manager.cc:1939] T 1dd29cdea6494a009e7b5e580bdf0cba P 9fbc6873d14f4e03b1fa9e455798a8cc: Deleting consensus metadata
I20260925 02:31:25.242023 11957 ts_tablet_manager.cc:1918] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:25.242127 11957 log.cc:1196] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/1dd29cdea6494a009e7b5e580bdf0cba
I20260925 02:31:25.242347 11957 ts_tablet_manager.cc:1939] T 1dd29cdea6494a009e7b5e580bdf0cba P 762c5149257e437ba541a9edbd1a7cbb: Deleting consensus metadata
I20260925 02:31:25.242430 11408 catalog_manager.cc:4841] TS 4fe0920b1a16434bacce34f7268e20cd (127.10.13.3:39491): tablet 1dd29cdea6494a009e7b5e580bdf0cba (table test-table [id=bf269fbb1ca0474aadd4029cfa4b290b]) successfully deleted
I20260925 02:31:25.242527 11411 catalog_manager.cc:4841] TS 9fbc6873d14f4e03b1fa9e455798a8cc (127.10.13.2:36837): tablet 1dd29cdea6494a009e7b5e580bdf0cba (table test-table [id=bf269fbb1ca0474aadd4029cfa4b290b]) successfully deleted
I20260925 02:31:25.242673 11409 catalog_manager.cc:4841] TS 762c5149257e437ba541a9edbd1a7cbb (127.10.13.1:41289): tablet 1dd29cdea6494a009e7b5e580bdf0cba (table test-table [id=bf269fbb1ca0474aadd4029cfa4b290b]) successfully deleted
Sep 25 02:31:25 dist-test-slave-q8mg krb5kdc[11379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:25.251019 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.241408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.10.13.1:41289)
0925 02:31:25.241528 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:25.241530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:25.241896 (+ 366us) server_negotiation.cc:407] Connection header received
0925 02:31:25.241998 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:25.241999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:25.242012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:25.242080 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:25.243027 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.243518 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.244271 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.244491 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.244572 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.247882 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:25.247898 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:25.247898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:25.247907 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:25.249725 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:25.250141 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:25.250148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:25.250148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:25.250212 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:25.250424 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:25.250425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:25.250426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:25.250593 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:25.250781 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:25.250820 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 02:31:25.250905 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260925 02:31:25.251308 11622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45090
I20260925 02:31:25.259059 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:25.252395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.10.13.62:40435)
0925 02:31:25.252452 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:25.252454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:25.252464 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:25.252540 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:25.252541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:25.252556 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:25.252613 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:25.253237 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.253804 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.254563 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:25.254782 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:25.254871 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:25.256153 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:25.256154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:25.256155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:25.256163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:25.257756 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:25.258136 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:25.258138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:25.258138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:25.258206 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:25.258459 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:25.258460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:25.258461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:25.258587 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:25.258720 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:25.258856 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 02:31:25.258950 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 02:31:25.259403 11424 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34498
I20260925 02:31:25.260149 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 11496
I20260925 02:31:25.268857 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 11637
I20260925 02:31:25.275662 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 11778
I20260925 02:31:25.281575 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 11392
2026-09-25T02:31:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2577 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:25 dist-test-slave-q8mg krb5kdc[12000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:25 dist-test-slave-q8mg krb5kdc[12000](info): set up 2 sockets
Sep 25 02:31:25 dist-test-slave-q8mg krb5kdc[12000](info): commencing operation
krb5kdc: starting...
W20260925 02:31:26.327225 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:26 dist-test-slave-q8mg krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:26.451845 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:41731
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:26.537962 12013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:26.538203 12013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:26.538239 12013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:26.539379 12013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:26.539445 12013 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:31:26.539466 12013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:26.539487 12013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:26.539520 12013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:26.539551 12013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:26.541050 12013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:41731
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:26.541342 12013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:26.541574 12013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:31:26.546627 12013 server_base.cc:1034] running on GCE node
W20260925 02:31:26.546594 12021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:26.546638 12019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:26.546594 12018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:26.547049 12013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:26.547329 12013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:26.548470 12013 hybrid_clock.cc:648] HybridClock initialized: now 1790303486548451 us; error 39 us; skew 500 ppm
Sep 25 02:31:26 dist-test-slave-q8mg krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303486, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:26.551178 12013 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:26.551590 12013 webserver.cc:460] Webserver started at http://127.10.13.62:39381/ using document root <none> and password file <none>
I20260925 02:31:26.551774 12013 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:26.551815 12013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:26.551903 12013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:26.553041 12013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "4bae0b765c034241bdf9f78b4e1dd531"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "688b328443a4a73ea182ec1d3a788dfb"
server_key_iv: "d5e62e1f63a26e766fa826e7ff40b549"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.553356 12013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "4bae0b765c034241bdf9f78b4e1dd531"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "688b328443a4a73ea182ec1d3a788dfb"
server_key_iv: "d5e62e1f63a26e766fa826e7ff40b549"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.554692 12013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:26.555428 12028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:26.555651 12013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:26.555727 12013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "4bae0b765c034241bdf9f78b4e1dd531"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "688b328443a4a73ea182ec1d3a788dfb"
server_key_iv: "d5e62e1f63a26e766fa826e7ff40b549"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.555784 12013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:26.566210 12013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:26.568054 12013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:26.568174 12013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:26.573148 12013 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:41731
I20260925 02:31:26.573263 12090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:41731 every 8 connection(s)
I20260925 02:31:26.573593 12013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:26.574373 12091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:26.576740 12091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: Bootstrap starting.
I20260925 02:31:26.576992 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 12013
I20260925 02:31:26.577096 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:26.577250 10292 external_mini_cluster.cc:1428] Setting key 42a118ae698e8d148ba8c6371052a7d1
I20260925 02:31:26.577483 12091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:26.577837 12091 log.cc:824] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:26.578536 12091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: No bootstrap required, opened a new log
I20260925 02:31:26.579901 12091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bae0b765c034241bdf9f78b4e1dd531" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41731 } }
I20260925 02:31:26.580065 12091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:26.580113 12091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bae0b765c034241bdf9f78b4e1dd531, State: Initialized, Role: FOLLOWER
I20260925 02:31:26.580240 12091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bae0b765c034241bdf9f78b4e1dd531" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41731 } }
I20260925 02:31:26.580333 12091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:26.580376 12091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:26.580427 12091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:26.580986 12091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bae0b765c034241bdf9f78b4e1dd531" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41731 } }
I20260925 02:31:26.581110 12091 leader_election.cc:302] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bae0b765c034241bdf9f78b4e1dd531; no voters:
I20260925 02:31:26.581296 12091 leader_election.cc:288] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:26.581331 12096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:26.581451 12096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [term 1 LEADER]: Becoming Leader. State: Replica: 4bae0b765c034241bdf9f78b4e1dd531, State: Running, Role: LEADER
I20260925 02:31:26.581566 12091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:26.581552 12096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bae0b765c034241bdf9f78b4e1dd531" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41731 } }
I20260925 02:31:26.582038 12096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bae0b765c034241bdf9f78b4e1dd531. Latest consensus state: current_term: 1 leader_uuid: "4bae0b765c034241bdf9f78b4e1dd531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bae0b765c034241bdf9f78b4e1dd531" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41731 } } }
I20260925 02:31:26.582046 12099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bae0b765c034241bdf9f78b4e1dd531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bae0b765c034241bdf9f78b4e1dd531" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41731 } } }
I20260925 02:31:26.582141 12096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:26.582226 12099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:26.582476 12106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:26.583030 12106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:26.584723 12106 catalog_manager.cc:1347] Generated new cluster ID: 4f540247da7c46e89ccd0fe47f2a8b38
I20260925 02:31:26.584777 12106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:31:26 dist-test-slave-q8mg krb5kdc[12000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:26.588552 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.577790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.10.13.62:41731)
0925 02:31:26.577988 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:26.577997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:26.578023 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:31:26.578457 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:26.578464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:26.578509 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:26.578666 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:26.579297 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.580715 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.581778 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.582380 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.582466 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.585349 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:26.585353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:26.585354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:26.585367 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:26.587051 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.587438 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.587442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.587442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.587527 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.587741 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.587742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.587742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.587970 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:26.588142 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:26.588236 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 02:31:26.588380 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260925 02:31:26.612987 12106 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:26.613777 12106 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:26.627069 12106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: Generated new TSK 0
I20260925 02:31:26.627319 12106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:26.664462 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41731
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:26.767094 12118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:26.767307 12118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:26.767333 12118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:26.768401 12118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:26.768432 12118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:26.768452 12118 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:31:26.768464 12118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:26.768481 12118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:26.769892 12118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:26.770088 12118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:26.770267 12118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:26.775377 12124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:26.775570 12118 server_base.cc:1034] running on GCE node
W20260925 02:31:26.775489 12123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:26.775357 12126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:26.775827 12118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:26.776036 12118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:26.777173 12118 hybrid_clock.cc:648] HybridClock initialized: now 1790303486777156 us; error 28 us; skew 500 ppm
Sep 25 02:31:26 dist-test-slave-q8mg krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303486, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:26.779759 12118 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:26.780282 12118 webserver.cc:460] Webserver started at http://127.10.13.1:43801/ using document root <none> and password file <none>
I20260925 02:31:26.780501 12118 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:26.780556 12118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:26.780668 12118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:26.781981 12118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "ed20eea09c914726ace4c3fee97ac351"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "449b9920be3c730bfcec6120440738be"
server_key_iv: "f125b5dad18c9c733681864c325c44a6"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.782303 12118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "ed20eea09c914726ace4c3fee97ac351"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "449b9920be3c730bfcec6120440738be"
server_key_iv: "f125b5dad18c9c733681864c325c44a6"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.783663 12118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:31:26.784330 12133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:26.784533 12118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:26.784626 12118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "ed20eea09c914726ace4c3fee97ac351"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "449b9920be3c730bfcec6120440738be"
server_key_iv: "f125b5dad18c9c733681864c325c44a6"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.784682 12118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:26.801697 12118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:26.803694 12118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:26.803808 12118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:26.804414 12118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:26.804495 12118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:26.804536 12118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:26.804582 12118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:26.811555 12118 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:44807
I20260925 02:31:26.811697 12249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:44807 every 8 connection(s)
I20260925 02:31:26.811959 12118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:31:26 dist-test-slave-q8mg krb5kdc[12000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303486, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:26.815155 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.805096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:54243 (local address 127.10.13.62:41731)
0925 02:31:26.805142 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:26.805144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:26.805350 (+ 206us) server_negotiation.cc:407] Connection header received
0925 02:31:26.805989 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:26.805990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:26.806008 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:26.806084 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:26.807041 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.807892 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.809539 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.809838 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.809933 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.812664 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:26.812665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:26.812666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:26.812677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:26.813957 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.814352 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.814353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.814353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.814405 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.814626 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.814627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.814627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.814733 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:26.814784 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:26.815017 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:31:26.815084 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:31:26.815348 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.805040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41731 (local address 127.10.13.1:54243)
0925 02:31:26.805224 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:26.805262 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:26.805835 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:26.806179 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:26.806183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:26.806597 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:26.806915 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:26.806935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.808003 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.808004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:26.809435 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:26.809438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.809948 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.809949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:26.810121 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.810722 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:26.810734 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:26.812571 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:26.814030 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:26.814064 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:26.814066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:26.814285 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:26.814463 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:26.814463 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:26.814464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:26.814562 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:26.814787 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:26.814791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:26.814906 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 02:31:26.814986 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:31:26.815141 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260925 02:31:26.815902 12250 heartbeater.cc:347] Connected to a master server at 127.10.13.62:41731
I20260925 02:31:26.815987 12250 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:26.816193 12250 heartbeater.cc:511] Master 127.10.13.62:41731 requested a full tablet report, sending...
I20260925 02:31:26.816780 12044 ts_manager.cc:194] Registered new tserver with Master: ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807)
I20260925 02:31:26.817788 12044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:54243
I20260925 02:31:26.820678 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 12118
I20260925 02:31:26.820775 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:26.820997 10292 external_mini_cluster.cc:1428] Setting key 6eb1b30a94165921d6c64b0a6e2d1294
I20260925 02:31:26.822999 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.816972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41731 (local address 127.10.13.1:52761)
0925 02:31:26.817190 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:26.817202 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:26.817268 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:26.817385 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:26.817386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:26.817575 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:26.817633 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:26.817634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.818441 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.818442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:26.819102 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:26.819111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.819438 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.819438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:26.819494 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.819860 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:26.819864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:26.820151 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:26.822068 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:26.822069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:26.822069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:26.822231 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:26.822436 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:26.822437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:26.822437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:26.822488 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:26.822765 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:26.822766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:26.822804 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:31:26.822856 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:31:26.822932 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":143,"threads_started":1}
I20260925 02:31:26.823146 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.817034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:52761 (local address 127.10.13.62:41731)
0925 02:31:26.817094 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:26.817096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:26.817267 (+ 171us) server_negotiation.cc:407] Connection header received
0925 02:31:26.817322 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:26.817323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:26.817332 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:26.817373 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:26.817690 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.818348 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.819190 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.819371 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.819427 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.820228 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:26.820229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:26.820230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:26.820236 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:26.821962 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.822301 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.822303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.822303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.822366 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.822548 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.822549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.822549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.822694 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:26.822759 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:26.822903 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:31:26.822987 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:26.848151 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41731
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:26.931375 12259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:26.931634 12259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:26.931679 12259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:26.932890 12259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:26.932960 12259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:26.932993 12259 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:31:26.933019 12259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:26.933065 12259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:26.934581 12259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:26.934859 12259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:26.935111 12259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:26.940292 12265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:26.940292 12264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:26.940500 12259 server_base.cc:1034] running on GCE node
W20260925 02:31:26.940300 12267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:26.940851 12259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:26.941141 12259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:26.942286 12259 hybrid_clock.cc:648] HybridClock initialized: now 1790303486942272 us; error 30 us; skew 500 ppm
Sep 25 02:31:26 dist-test-slave-q8mg krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303486, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:26.945226 12259 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:26.945660 12259 webserver.cc:460] Webserver started at http://127.10.13.2:44335/ using document root <none> and password file <none>
I20260925 02:31:26.945940 12259 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:26.946008 12259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:26.946162 12259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:26.947253 12259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "8e5ae2e73e104faab3318a8d67cf2420"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "f8ecd5f43ac079f81a2a0db0d942f6df"
server_key_iv: "bb605bf9e3d4c796ca517cfa343d0739"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.947554 12259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "8e5ae2e73e104faab3318a8d67cf2420"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "f8ecd5f43ac079f81a2a0db0d942f6df"
server_key_iv: "bb605bf9e3d4c796ca517cfa343d0739"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.948973 12259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:31:26.949769 12274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:26.949967 12259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:26.950045 12259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "8e5ae2e73e104faab3318a8d67cf2420"
format_stamp: "Formatted at 2026-09-25 02:31:26 on dist-test-slave-q8mg"
server_key: "f8ecd5f43ac079f81a2a0db0d942f6df"
server_key_iv: "bb605bf9e3d4c796ca517cfa343d0739"
server_key_version: "encryptionkey@0"
I20260925 02:31:26.950100 12259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:26.963651 12259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:26.965715 12259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:26.965835 12259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:26.966331 12259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:26.966370 12259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:26.966403 12259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:26.966455 12259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:26.973145 12259 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:40645
I20260925 02:31:26.973212 12390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:40645 every 8 connection(s)
I20260925 02:31:26.973531 12259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:31:26 dist-test-slave-q8mg krb5kdc[12000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303486, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:26.977011 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.966992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:51599 (local address 127.10.13.62:41731)
0925 02:31:26.967050 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:26.967052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:26.967316 (+ 264us) server_negotiation.cc:407] Connection header received
0925 02:31:26.967812 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:26.967813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:26.967828 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:26.967912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:26.968777 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.969647 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.971202 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.971453 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.971538 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.974330 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:26.974332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:26.974332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:26.974341 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:26.975537 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.975950 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.975952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.975952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.976021 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.976296 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.976296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.976297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.976424 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:26.976498 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:26.976802 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:31:26.976918 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:31:26.977221 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.966990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41731 (local address 127.10.13.2:51599)
0925 02:31:26.967182 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:26.967219 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:26.967707 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:26.967965 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:26.967968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:26.968351 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:26.968665 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:26.968672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.969737 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.969739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:26.971119 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:26.971123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.971553 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.971554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:26.971755 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.972455 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:26.972469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:26.974217 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:26.975629 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:26.975637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:26.975640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:26.975869 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:26.976100 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:26.976101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:26.976101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:26.976219 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:26.976498 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:26.976505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:26.976658 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 02:31:26.976765 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:31:26.976970 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260925 02:31:26.977795 12391 heartbeater.cc:347] Connected to a master server at 127.10.13.62:41731
I20260925 02:31:26.977903 12391 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:26.978096 12391 heartbeater.cc:511] Master 127.10.13.62:41731 requested a full tablet report, sending...
I20260925 02:31:26.978493 12045 ts_manager.cc:194] Registered new tserver with Master: 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645)
I20260925 02:31:26.978926 12045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:51599
I20260925 02:31:26.983198 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 12259
I20260925 02:31:26.983330 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:26.983536 10292 external_mini_cluster.cc:1428] Setting key d2c6ffde10ea53d23000279af368dcf5
I20260925 02:31:26.984449 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.978880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:46207 (local address 127.10.13.62:41731)
0925 02:31:26.978928 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:26.978930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:26.979072 (+ 142us) server_negotiation.cc:407] Connection header received
0925 02:31:26.979168 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:26.979169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:26.979179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:26.979228 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:26.979534 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.980008 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.980730 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.981008 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.981091 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.981971 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:26.981972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:26.981973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:26.981980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:26.983201 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.983614 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.983615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.983615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.983674 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:26.983884 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:26.983884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:26.983885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:26.984041 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:26.984121 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:26.984282 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 02:31:26.984358 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:26.984463 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:26.978816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41731 (local address 127.10.13.2:46207)
0925 02:31:26.978987 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:26.979002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:26.979097 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:26.979230 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:26.979231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:26.979409 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:26.979460 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:26.979461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.980113 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.980114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:26.980647 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:26.980648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:26.981094 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:26.981095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:26.981231 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:26.981618 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:26.981621 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:26.981880 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:26.983324 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:26.983325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:26.983325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:26.983542 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:26.983746 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:26.983747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:26.983748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:26.983807 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:26.984120 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:26.984121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:26.984168 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:31:26.984256 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:31:26.984360 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:27.009665 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41731
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:27.095053 12400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:27.095299 12400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:27.095343 12400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:27.096511 12400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:27.096580 12400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:27.096624 12400 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:31:27.096671 12400 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:27.096701 12400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:27.098256 12400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:27.098532 12400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:27.098801 12400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:27.103991 12406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:27.104113 12400 server_base.cc:1034] running on GCE node
W20260925 02:31:27.104058 12405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:27.103991 12408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:27.104490 12400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:27.104745 12400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:27.105885 12400 hybrid_clock.cc:648] HybridClock initialized: now 1790303487105875 us; error 61 us; skew 500 ppm
Sep 25 02:31:27 dist-test-slave-q8mg krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303487, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:27.108414 12400 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:27.108918 12400 webserver.cc:460] Webserver started at http://127.10.13.3:38977/ using document root <none> and password file <none>
I20260925 02:31:27.109190 12400 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:27.109239 12400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:27.109412 12400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:27.110514 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "3c4f448843e34f68aee74193e07520b3"
format_stamp: "Formatted at 2026-09-25 02:31:27 on dist-test-slave-q8mg"
server_key: "38cc2be83002ea7092dec6c2b4bfa7bf"
server_key_iv: "9089157f988e26b6c17d3cc429b9dc7e"
server_key_version: "encryptionkey@0"
I20260925 02:31:27.110880 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "3c4f448843e34f68aee74193e07520b3"
format_stamp: "Formatted at 2026-09-25 02:31:27 on dist-test-slave-q8mg"
server_key: "38cc2be83002ea7092dec6c2b4bfa7bf"
server_key_iv: "9089157f988e26b6c17d3cc429b9dc7e"
server_key_version: "encryptionkey@0"
I20260925 02:31:27.112265 12400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:27.113190 12415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:27.113389 12400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:27.113483 12400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "3c4f448843e34f68aee74193e07520b3"
format_stamp: "Formatted at 2026-09-25 02:31:27 on dist-test-slave-q8mg"
server_key: "38cc2be83002ea7092dec6c2b4bfa7bf"
server_key_iv: "9089157f988e26b6c17d3cc429b9dc7e"
server_key_version: "encryptionkey@0"
I20260925 02:31:27.113572 12400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:27.134331 12400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:27.136530 12400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:27.136736 12400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:27.137389 12400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:27.137437 12400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:27.137462 12400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:27.137517 12400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:31:27 dist-test-slave-q8mg krb5kdc[12000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303487, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:27.145488 12400 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:37025
I20260925 02:31:27.145627 12531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:37025 every 8 connection(s)
I20260925 02:31:27.145958 12400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:27.148653 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.137876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:59777 (local address 127.10.13.62:41731)
0925 02:31:27.137951 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.137954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.138190 (+ 236us) server_negotiation.cc:407] Connection header received
0925 02:31:27.138679 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.138681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.138696 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.138795 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:27.139631 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.140568 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.141779 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.142170 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.142255 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.145675 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:27.145677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:27.145677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:27.145686 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:27.147387 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:27.147814 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:27.147815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:27.147815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:27.147868 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:27.148071 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:27.148072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:27.148072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:27.148189 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:27.148241 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.148508 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.148581 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:31:27.148943 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.137846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41731 (local address 127.10.13.3:59777)
0925 02:31:27.138038 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.138080 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.138572 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.138848 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.138851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.139205 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:27.139520 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.139528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.140699 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.140700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.141673 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.141676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.142293 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.142294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.142410 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.142967 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:27.142976 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:27.145550 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:27.147470 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:27.147477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:27.147479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:27.147742 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:27.147911 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:27.147911 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:27.147911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:27.148011 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:27.148255 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:27.148261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:27.148389 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.148481 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.148625 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
I20260925 02:31:27.149407 12532 heartbeater.cc:347] Connected to a master server at 127.10.13.62:41731
I20260925 02:31:27.149482 12532 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:27.149708 12532 heartbeater.cc:511] Master 127.10.13.62:41731 requested a full tablet report, sending...
I20260925 02:31:27.150147 12044 ts_manager.cc:194] Registered new tserver with Master: 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025)
I20260925 02:31:27.150743 12044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:59777
I20260925 02:31:27.155754 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 12400
I20260925 02:31:27.155856 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:27.156072 10292 external_mini_cluster.cc:1428] Setting key 12e601c21a28c05ab8f4ece89e958d95
I20260925 02:31:27.157171 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:31:27.157349 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.150619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:37117 (local address 127.10.13.62:41731)
0925 02:31:27.150656 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.150658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.150763 (+ 105us) server_negotiation.cc:407] Connection header received
0925 02:31:27.150853 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.150853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.150880 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.150939 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:27.151470 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.152000 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.153000 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.153198 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.153258 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.154270 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:27.154271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:27.154272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:27.154279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:27.155580 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:27.156300 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:27.156301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:27.156302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:27.156370 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:27.156705 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:27.156706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:27.156707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:27.156844 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:27.156952 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.157157 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.157260 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:31:27.157358 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.150542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41731 (local address 127.10.13.3:37117)
0925 02:31:27.150702 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.150713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.150773 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.151087 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.151088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.151271 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:27.151371 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.151373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.152108 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.152109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.152786 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.152789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.153265 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.153266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.153331 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.153840 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:27.153845 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:27.154162 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:27.155951 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:27.155952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:27.155952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:27.156220 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:27.156552 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:27.156553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:27.156554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:27.156642 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:27.156955 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:27.156957 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:27.157024 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.157126 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.157247 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
Sep 25 02:31:27 dist-test-slave-q8mg krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:31:27 dist-test-slave-q8mg krb5kdc[12000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:27.180604 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.173376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.10.13.62:41731)
0925 02:31:27.173446 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.173448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.173457 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:27.173518 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.173519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.173528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.173579 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:27.174182 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.174670 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.175473 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.175655 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.175704 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.178012 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:27.178014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:27.178014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:27.178021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:27.179517 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:27.179853 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:27.179855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:27.179855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:27.179906 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:27.180179 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:27.180180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:27.180180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:27.180288 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:27.180370 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.180473 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.180534 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:31:27.182402 12044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:27.184142 12044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:27.191514 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.187071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.10.13.3:37025)
0925 02:31:27.187318 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.187322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.187331 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:27.187452 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.187455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.187473 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.187550 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:27.187783 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.188966 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.190370 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.191053 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.191130 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.191179 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.191354 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.191408 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
I20260925 02:31:27.191514 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.186893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:40645 (local address 127.0.0.1:39898)
0925 02:31:27.187477 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.187486 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.187525 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.187798 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.187799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.187801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:27.187835 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.187836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.189149 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.189150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.190141 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.190144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.191139 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.191139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.191232 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.191247 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.191329 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.191403 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":33,"thread_start_us":88,"threads_started":1}
I20260925 02:31:27.191515 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.186893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:37025 (local address 127.0.0.1:59400)
0925 02:31:27.187268 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.187279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.187335 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.187653 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.187654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.187657 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:27.187718 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.187719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.189052 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.189052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.190280 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.190283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.191137 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.191139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.191231 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.191247 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.191329 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.191403 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":35,"thread_start_us":90,"threads_started":1}
I20260925 02:31:27.192158 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.187091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.10.13.2:40645)
0925 02:31:27.187499 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.187503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.187516 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:31:27.187634 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.187636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.187650 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.187719 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:27.187957 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.189071 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.190232 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.191012 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.191098 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.191167 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.192000 (+ 833us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.192045 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":80,"threads_started":1}
I20260925 02:31:27.192328 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.186787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:44807 (local address 127.0.0.1:37544)
0925 02:31:27.187058 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.187114 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.187183 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.187631 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.187634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.187636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:27.187717 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.187719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.188891 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.188892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.190294 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.190296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.192144 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.192145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.192205 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.192208 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.192251 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.192281 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":8,"thread_start_us":73,"threads_started":1}
I20260925 02:31:27.192408 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.187097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.10.13.1:44807)
0925 02:31:27.187297 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.187302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.187312 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:27.187357 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.187359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.187376 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.187828 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:27.187903 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.188783 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.191217 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.192030 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.192140 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.192203 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.192266 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.192303 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260925 02:31:27.193208 12325 tablet_service.cc:1467] Processing CreateTablet for tablet 15c596ca24de4974a97c705f5353399a (DEFAULT_TABLE table=test-table [id=4a8331d04a4d482d941a0e45613ad253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:27.193183 12466 tablet_service.cc:1467] Processing CreateTablet for tablet 15c596ca24de4974a97c705f5353399a (DEFAULT_TABLE table=test-table [id=4a8331d04a4d482d941a0e45613ad253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:27.193186 12183 tablet_service.cc:1467] Processing CreateTablet for tablet 15c596ca24de4974a97c705f5353399a (DEFAULT_TABLE table=test-table [id=4a8331d04a4d482d941a0e45613ad253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:27.193552 12325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c596ca24de4974a97c705f5353399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.193548 12183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c596ca24de4974a97c705f5353399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.193554 12466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c596ca24de4974a97c705f5353399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.196633 12555 tablet_bootstrap.cc:492] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Bootstrap starting.
I20260925 02:31:27.197003 12556 tablet_bootstrap.cc:492] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Bootstrap starting.
I20260925 02:31:27.197252 12557 tablet_bootstrap.cc:492] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Bootstrap starting.
I20260925 02:31:27.197460 12555 tablet_bootstrap.cc:654] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.197836 12557 tablet_bootstrap.cc:654] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.197875 12555 log.cc:824] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:27.198091 12556 tablet_bootstrap.cc:654] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.198205 12557 log.cc:824] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:27.198467 12556 log.cc:824] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:27.198943 12557 tablet_bootstrap.cc:492] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: No bootstrap required, opened a new log
I20260925 02:31:27.199019 12557 ts_tablet_manager.cc:1397] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:27.199223 12555 tablet_bootstrap.cc:492] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: No bootstrap required, opened a new log
I20260925 02:31:27.199294 12556 tablet_bootstrap.cc:492] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: No bootstrap required, opened a new log
I20260925 02:31:27.199306 12555 ts_tablet_manager.cc:1397] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:31:27.199360 12556 ts_tablet_manager.cc:1397] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:31:27.200695 12555 raft_consensus.cc:348] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.200860 12555 raft_consensus.cc:374] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.200920 12555 raft_consensus.cc:729] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5ae2e73e104faab3318a8d67cf2420, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.201066 12555 consensus_queue.cc:260] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.201138 12557 raft_consensus.cc:348] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.201308 12557 raft_consensus.cc:374] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.201350 12557 raft_consensus.cc:729] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed20eea09c914726ace4c3fee97ac351, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.201385 12555 ts_tablet_manager.cc:1428] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:31:27.201529 12391 heartbeater.cc:503] Master 127.10.13.62:41731 was elected leader, sending a full tablet report...
I20260925 02:31:27.201429 12556 raft_consensus.cc:348] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.201493 12557 consensus_queue.cc:260] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.201565 12556 raft_consensus.cc:374] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.201607 12556 raft_consensus.cc:729] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f448843e34f68aee74193e07520b3, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.201725 12556 consensus_queue.cc:260] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.201792 12250 heartbeater.cc:503] Master 127.10.13.62:41731 was elected leader, sending a full tablet report...
I20260925 02:31:27.201802 12557 ts_tablet_manager.cc:1428] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:27.201977 12556 ts_tablet_manager.cc:1428] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 02:31:27.202135 12532 heartbeater.cc:503] Master 127.10.13.62:41731 was elected leader, sending a full tablet report...
W20260925 02:31:27.224267 12392 tablet.cc:2369] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:27.242822 12561 raft_consensus.cc:482] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:27.243027 12561 raft_consensus.cc:504] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.243525 12561 leader_election.cc:288] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807), 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025)
I20260925 02:31:27.246971 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.243795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:52131 (local address 127.10.13.3:37025)
0925 02:31:27.243845 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.243847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.244009 (+ 162us) server_negotiation.cc:407] Connection header received
0925 02:31:27.244081 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.244082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.244093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.244165 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:27.244331 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.244900 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.246244 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.246607 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.246665 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.246673 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.246853 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.246899 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:31:27.246989 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.243738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:37025 (local address 127.10.13.2:52131)
0925 02:31:27.243928 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.243938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.244024 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.244183 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.244184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.244185 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:27.244254 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.244256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.244983 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.244984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.246159 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.246161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.246691 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.246692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.246768 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.246773 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.246828 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.246870 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":25,"thread_start_us":115,"threads_started":1}
I20260925 02:31:27.247071 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.243793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:40103 (local address 127.10.13.1:44807)
0925 02:31:27.243849 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.243851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.243862 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:27.243928 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.243929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.243940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.243991 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:27.244165 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.244784 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.246289 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.246688 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.246759 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.246770 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.246942 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.246988 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:31:27.247278 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.243738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:44807 (local address 127.10.13.2:40103)
0925 02:31:27.243809 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.243820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.243868 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.243992 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.243993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.243994 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:27.244078 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.244080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.244897 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.244898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.246158 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.246161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.246756 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.246757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.246859 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.246862 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.246916 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.246954 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":42}
I20260925 02:31:27.247466 12486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c596ca24de4974a97c705f5353399a" candidate_uuid: "8e5ae2e73e104faab3318a8d67cf2420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4f448843e34f68aee74193e07520b3" is_pre_election: true
I20260925 02:31:27.247656 12486 raft_consensus.cc:2454] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e5ae2e73e104faab3318a8d67cf2420 in term 0.
I20260925 02:31:27.247679 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c596ca24de4974a97c705f5353399a" candidate_uuid: "8e5ae2e73e104faab3318a8d67cf2420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20eea09c914726ace4c3fee97ac351" is_pre_election: true
I20260925 02:31:27.247802 12204 raft_consensus.cc:2454] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e5ae2e73e104faab3318a8d67cf2420 in term 0.
I20260925 02:31:27.247924 12275 leader_election.cc:302] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f448843e34f68aee74193e07520b3, 8e5ae2e73e104faab3318a8d67cf2420; no voters:
I20260925 02:31:27.248098 12561 raft_consensus.cc:2781] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:27.248159 12561 raft_consensus.cc:482] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:27.248189 12561 raft_consensus.cc:3037] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.249205 12561 raft_consensus.cc:504] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.249368 12561 leader_election.cc:288] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [CANDIDATE]: Term 1 election: Requested vote from peers ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807), 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025)
I20260925 02:31:27.249516 12486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c596ca24de4974a97c705f5353399a" candidate_uuid: "8e5ae2e73e104faab3318a8d67cf2420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4f448843e34f68aee74193e07520b3"
I20260925 02:31:27.249522 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c596ca24de4974a97c705f5353399a" candidate_uuid: "8e5ae2e73e104faab3318a8d67cf2420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20eea09c914726ace4c3fee97ac351"
I20260925 02:31:27.249612 12486 raft_consensus.cc:3037] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.249609 12204 raft_consensus.cc:3037] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.250438 12486 raft_consensus.cc:2454] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e5ae2e73e104faab3318a8d67cf2420 in term 1.
I20260925 02:31:27.250578 12204 raft_consensus.cc:2454] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e5ae2e73e104faab3318a8d67cf2420 in term 1.
I20260925 02:31:27.250628 12275 leader_election.cc:302] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f448843e34f68aee74193e07520b3, 8e5ae2e73e104faab3318a8d67cf2420; no voters:
I20260925 02:31:27.250721 12561 raft_consensus.cc:2781] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:27.250908 12561 raft_consensus.cc:686] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 LEADER]: Becoming Leader. State: Replica: 8e5ae2e73e104faab3318a8d67cf2420, State: Running, Role: LEADER
I20260925 02:31:27.251019 12561 consensus_queue.cc:237] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } }
I20260925 02:31:27.251825 12044 catalog_manager.cc:5495] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2). New cstate: current_term: 1 leader_uuid: "8e5ae2e73e104faab3318a8d67cf2420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } health_report { overall_health: HEALTHY } } }
Sep 25 02:31:27 dist-test-slave-q8mg krb5kdc[12000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303486, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:27.293051 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.285786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.10.13.62:41731)
0925 02:31:27.285883 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.285886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.286070 (+ 184us) server_negotiation.cc:407] Connection header received
0925 02:31:27.286102 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.286109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.286119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.286182 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:27.286679 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.287238 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.287990 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.288200 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.288259 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.290480 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:27.290482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:27.290482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:27.290489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:27.291863 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:27.292274 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:27.292275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:27.292276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:27.292327 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:27.292545 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:27.292546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:27.292547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:27.292656 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:27.292706 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.292861 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.292980 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:31:27.293324 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.285727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41731 (local address 127.0.0.1:43642)
0925 02:31:27.285915 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.285933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.286011 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.286199 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.286200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.286543 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:27.286590 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.286593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.287319 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.287320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.287883 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.287885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.288256 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.288257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.288312 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.288852 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:27.288857 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:27.290368 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:27.291945 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:27.291952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:27.291954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:27.292206 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:27.292411 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:27.292412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:27.292412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:27.292480 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:27.292715 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:27.292727 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:27.292774 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.292828 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.293176 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":67,"thread_start_us":74,"threads_started":1}
I20260925 02:31:27.295560 12044 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:43642:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:31:27.295831 12044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:27.297597 12325 tablet_service.cc:1467] Processing CreateTablet for tablet cf7a33581f064657943b0c2d1294c93e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=86f93dcb4f264581bb267d9057bd4c20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:27.297592 12466 tablet_service.cc:1467] Processing CreateTablet for tablet cf7a33581f064657943b0c2d1294c93e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=86f93dcb4f264581bb267d9057bd4c20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:27.297592 12183 tablet_service.cc:1467] Processing CreateTablet for tablet cf7a33581f064657943b0c2d1294c93e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=86f93dcb4f264581bb267d9057bd4c20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:27.297782 12183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7a33581f064657943b0c2d1294c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.297781 12466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7a33581f064657943b0c2d1294c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.297782 12325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7a33581f064657943b0c2d1294c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.299078 12555 tablet_bootstrap.cc:492] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: Bootstrap starting.
I20260925 02:31:27.299343 12557 tablet_bootstrap.cc:492] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: Bootstrap starting.
I20260925 02:31:27.299564 12555 tablet_bootstrap.cc:654] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.299998 12557 tablet_bootstrap.cc:654] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.300239 12555 tablet_bootstrap.cc:492] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: No bootstrap required, opened a new log
I20260925 02:31:27.300304 12555 ts_tablet_manager.cc:1397] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:27.300392 12556 tablet_bootstrap.cc:492] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: Bootstrap starting.
I20260925 02:31:27.300472 12555 raft_consensus.cc:348] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.300580 12555 raft_consensus.cc:374] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.300611 12555 raft_consensus.cc:729] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5ae2e73e104faab3318a8d67cf2420, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.300661 12555 consensus_queue.cc:260] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.300760 12555 ts_tablet_manager.cc:1428] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:27.300834 12557 tablet_bootstrap.cc:492] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: No bootstrap required, opened a new log
I20260925 02:31:27.300940 12557 ts_tablet_manager.cc:1397] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:27.300923 12561 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.300999 12561 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:27.301064 12556 tablet_bootstrap.cc:654] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.301136 12557 raft_consensus.cc:348] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.301244 12557 raft_consensus.cc:374] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.301270 12557 raft_consensus.cc:729] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed20eea09c914726ace4c3fee97ac351, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.301328 12557 consensus_queue.cc:260] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.301434 12557 ts_tablet_manager.cc:1428] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:27.301553 12562 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.301723 12562 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:27.302328 12556 tablet_bootstrap.cc:492] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: No bootstrap required, opened a new log
I20260925 02:31:27.302371 12556 ts_tablet_manager.cc:1397] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:27.302479 12556 raft_consensus.cc:348] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.302528 12556 raft_consensus.cc:374] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.302544 12556 raft_consensus.cc:729] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f448843e34f68aee74193e07520b3, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.302603 12556 consensus_queue.cc:260] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.302734 12556 ts_tablet_manager.cc:1428] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:27.302834 12563 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.302930 12563 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 02:31:27.313176 12251 tablet.cc:2369] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:27.362030 12563 raft_consensus.cc:482] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:27.362174 12563 raft_consensus.cc:504] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.362619 12563 leader_election.cc:288] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807), 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645)
I20260925 02:31:27.366642 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.362952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:34111 (local address 127.10.13.1:44807)
0925 02:31:27.363015 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.363017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.363028 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:27.363061 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.363062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.363072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.363143 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:27.363317 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.364096 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.365665 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.366246 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.366326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.366337 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.366521 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.366568 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:31:27.366642 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.362938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:53899 (local address 127.10.13.2:40645)
0925 02:31:27.363015 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.363018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.363460 (+ 442us) server_negotiation.cc:407] Connection header received
0925 02:31:27.363492 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.363493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.363506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.363583 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:27.363727 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.364344 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.365665 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.366246 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.366326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.366337 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.366521 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.366572 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:31:27.366657 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.362829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:44807 (local address 127.10.13.3:34111)
0925 02:31:27.362908 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.362922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.362987 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.363148 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.363149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.363151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:27.363233 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.363235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.364206 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.364207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.365549 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.365552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.366338 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.366339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.366427 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.366434 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.366494 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.366549 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":41}
I20260925 02:31:27.366676 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.362829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:40645 (local address 127.10.13.3:53899)
0925 02:31:27.363377 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.363393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.363440 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.363590 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.363591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.363592 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:27.363657 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.363658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.364422 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.364423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.365562 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.365564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.366354 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.366354 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.366427 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.366434 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.366494 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.366549 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":351,"thread_start_us":114,"threads_started":1}
I20260925 02:31:27.366981 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7a33581f064657943b0c2d1294c93e" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20eea09c914726ace4c3fee97ac351" is_pre_election: true
I20260925 02:31:27.367128 12204 raft_consensus.cc:2454] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 0.
I20260925 02:31:27.367234 12326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7a33581f064657943b0c2d1294c93e" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ae2e73e104faab3318a8d67cf2420" is_pre_election: true
I20260925 02:31:27.367385 12326 raft_consensus.cc:2454] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 0.
I20260925 02:31:27.367352 12417 leader_election.cc:302] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f448843e34f68aee74193e07520b3, ed20eea09c914726ace4c3fee97ac351; no voters:
I20260925 02:31:27.367501 12563 raft_consensus.cc:2781] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:27.367569 12563 raft_consensus.cc:482] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:27.367609 12563 raft_consensus.cc:3037] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.368314 12563 raft_consensus.cc:504] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.368472 12563 leader_election.cc:288] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 election: Requested vote from peers ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807), 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645)
I20260925 02:31:27.368624 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7a33581f064657943b0c2d1294c93e" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20eea09c914726ace4c3fee97ac351"
I20260925 02:31:27.368631 12326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7a33581f064657943b0c2d1294c93e" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ae2e73e104faab3318a8d67cf2420"
I20260925 02:31:27.368701 12204 raft_consensus.cc:3037] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.368727 12326 raft_consensus.cc:3037] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.369364 12204 raft_consensus.cc:2454] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 1.
I20260925 02:31:27.369542 12417 leader_election.cc:302] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f448843e34f68aee74193e07520b3, ed20eea09c914726ace4c3fee97ac351; no voters:
I20260925 02:31:27.369647 12563 raft_consensus.cc:2781] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:27.369664 12326 raft_consensus.cc:2454] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 1.
I20260925 02:31:27.369829 12563 raft_consensus.cc:686] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4f448843e34f68aee74193e07520b3, State: Running, Role: LEADER
I20260925 02:31:27.369935 12563 consensus_queue.cc:237] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.370426 12577 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: TxnStatusTablet state changed. Reason: New leader 3c4f448843e34f68aee74193e07520b3. Latest consensus state: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.370522 12577 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:27.370718 12579 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:31:27.370834 12579 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:31:27.370769 12045 catalog_manager.cc:5495] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4f448843e34f68aee74193e07520b3 (127.10.13.3). New cstate: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } health_report { overall_health: HEALTHY } } }
I20260925 02:31:27.377904 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.374804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.10.13.3:37025)
0925 02:31:27.374843 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.374845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.374978 (+ 133us) server_negotiation.cc:407] Connection header received
0925 02:31:27.375018 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.375019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.375035 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.375097 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:27.375276 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.375740 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.376973 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.377251 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.377318 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.377539 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.377649 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.377715 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.377807 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.377832 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:31:27.377976 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.374754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:37025 (local address 127.0.0.1:59414)
0925 02:31:27.374848 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.374860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.374915 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.375131 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.375133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.375134 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:27.375190 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.375192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.375823 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.375824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.376833 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.376836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.377336 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.377338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.377418 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.377440 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.377720 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.377742 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.377804 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.377865 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:31:27.379927 12204 raft_consensus.cc:1261] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Refusing update from remote peer 3c4f448843e34f68aee74193e07520b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:27.379964 12326 raft_consensus.cc:1261] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Refusing update from remote peer 3c4f448843e34f68aee74193e07520b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:27.380302 12563 consensus_queue.cc:1035] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:27.380502 12577 consensus_queue.cc:1035] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:27.381711 12561 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: TxnStatusTablet state changed. Reason: New leader 3c4f448843e34f68aee74193e07520b3. Latest consensus state: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.381806 12561 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:27.382431 12562 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: TxnStatusTablet state changed. Reason: New leader 3c4f448843e34f68aee74193e07520b3. Latest consensus state: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.382546 12562 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:27.383872 12577 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.383965 12577 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:27.384075 12577 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.384138 12577 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P 3c4f448843e34f68aee74193e07520b3: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:27.384579 12562 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.384696 12562 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P ed20eea09c914726ace4c3fee97ac351: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:27.384791 12585 mvcc.cc:205] Tried to move back new op lower bound from 7333083084305383424 to 7333083084267847680. Current Snapshot: MvccSnapshot[applied={T|T < 7333083084305383424}]
I20260925 02:31:27.385572 12561 tablet_replica.cc:439] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } } }
I20260925 02:31:27.385667 12561 tablet_replica.cc:442] T cf7a33581f064657943b0c2d1294c93e P 8e5ae2e73e104faab3318a8d67cf2420: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:27.389292 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.386378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.10.13.2:40645)
0925 02:31:27.386427 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.386430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.386474 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:31:27.386568 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.386569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.386584 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.386641 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:27.386787 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.387310 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.388442 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.388677 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.388731 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.388963 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.389063 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.389133 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.389196 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.389216 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:31:27.394330 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.391989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:50007 (local address 127.10.13.3:37025)
0925 02:31:27.392053 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.392055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.392066 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:27.392113 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.392114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.392126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.392183 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:27.392323 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.392791 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.393583 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.393790 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.393867 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.394058 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.394103 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.394168 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.394233 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.394254 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:31:27.394338 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.391904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:37025 (local address 127.10.13.2:50007)
0925 02:31:27.391980 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.391996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.392056 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.392187 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.392188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.392189 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:27.392257 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.392258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.392902 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.392903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.393492 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.393494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.393865 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.393866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.393943 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.393958 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.394170 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.394173 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.394226 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.394272 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:31:27.396840 12204 raft_consensus.cc:1261] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Refusing update from remote peer 8e5ae2e73e104faab3318a8d67cf2420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:27.396935 12486 raft_consensus.cc:1261] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Refusing update from remote peer 8e5ae2e73e104faab3318a8d67cf2420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 02:31:27.397053 12533 tablet.cc:2369] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:27.397173 12561 consensus_queue.cc:1035] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:27.397307 12565 consensus_queue.cc:1035] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:27.398593 12583 mvcc.cc:205] Tried to move back new op lower bound from 7333083084375347200 to 7333083083780718592. Current Snapshot: MvccSnapshot[applied={T|T < 7333083084375347200}]
I20260925 02:31:27.399039 12582 mvcc.cc:205] Tried to move back new op lower bound from 7333083084375347200 to 7333083083780718592. Current Snapshot: MvccSnapshot[applied={T|T < 7333083084375347200}]
I20260925 02:31:27.406395 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.403580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:57567 (local address 127.10.13.2:40645)
0925 02:31:27.403619 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.403621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.403662 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:31:27.403750 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.403751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.403764 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.403824 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:27.403969 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.404731 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.405573 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.405874 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.405943 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.406156 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.406194 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.406236 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.406315 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.406336 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:31:27.406435 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.403540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:40645 (local address 127.10.13.3:57567)
0925 02:31:27.403617 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.403628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.403677 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.403828 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.403830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.403831 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:27.403905 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.403906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.404819 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.404820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.405482 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.405485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.405981 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.405982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.406051 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.406070 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.406256 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.406258 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.406306 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.406354 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:31:27.416903 12045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:27.417035 12045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:27.417936 12045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: Sending DeleteTablet for 3 replicas of tablet 15c596ca24de4974a97c705f5353399a
I20260925 02:31:27.418310 12325 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c596ca24de4974a97c705f5353399a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:27 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39898
I20260925 02:31:27.418704 12183 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c596ca24de4974a97c705f5353399a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:27 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:37544
I20260925 02:31:27.418785 12599 tablet_replica.cc:330] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: stopping tablet replica
I20260925 02:31:27.418893 12599 raft_consensus.cc:2229] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:27.418992 12599 raft_consensus.cc:2258] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:27.419034 12466 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c596ca24de4974a97c705f5353399a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:27 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:59400
I20260925 02:31:27.419478 12599 ts_tablet_manager.cc:1905] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:27.419507 12045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:27.419667 12045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:27.421211 12325 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed21d8ee0045259882b3b85d2450e7 (DEFAULT_TABLE table=test-table [id=525c673a52a24905ada9020644c05b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:27.421320 12325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed21d8ee0045259882b3b85d2450e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.421737 12466 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed21d8ee0045259882b3b85d2450e7 (DEFAULT_TABLE table=test-table [id=525c673a52a24905ada9020644c05b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:27.421860 12466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed21d8ee0045259882b3b85d2450e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.422561 12555 tablet_bootstrap.cc:492] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: Bootstrap starting.
I20260925 02:31:27.423084 12555 tablet_bootstrap.cc:654] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.423177 12556 tablet_bootstrap.cc:492] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: Bootstrap starting.
I20260925 02:31:27.423596 12556 tablet_bootstrap.cc:654] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.423569 12183 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed21d8ee0045259882b3b85d2450e7 (DEFAULT_TABLE table=test-table [id=525c673a52a24905ada9020644c05b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:27.423692 12183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed21d8ee0045259882b3b85d2450e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:27.424290 12600 tablet_replica.cc:330] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: stopping tablet replica
I20260925 02:31:27.424465 12600 raft_consensus.cc:2229] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:27.424582 12600 raft_consensus.cc:2258] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:27.425047 12557 tablet_bootstrap.cc:492] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: Bootstrap starting.
I20260925 02:31:27.425210 12601 tablet_replica.cc:330] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: stopping tablet replica
I20260925 02:31:27.425314 12600 ts_tablet_manager.cc:1905] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:27.425436 12557 tablet_bootstrap.cc:654] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:27.425901 12601 raft_consensus.cc:2229] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:27.426169 12599 ts_tablet_manager.cc:1918] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:27.426251 12599 log.cc:1196] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/15c596ca24de4974a97c705f5353399a
I20260925 02:31:27.426378 12601 raft_consensus.cc:2258] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:27.426614 12599 ts_tablet_manager.cc:1939] T 15c596ca24de4974a97c705f5353399a P 8e5ae2e73e104faab3318a8d67cf2420: Deleting consensus metadata
I20260925 02:31:27.427062 12032 catalog_manager.cc:4841] TS 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645): tablet 15c596ca24de4974a97c705f5353399a (table test-table [id=4a8331d04a4d482d941a0e45613ad253]) successfully deleted
I20260925 02:31:27.427907 12601 ts_tablet_manager.cc:1905] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:27.428162 12600 ts_tablet_manager.cc:1918] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:27.428287 12600 log.cc:1196] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/15c596ca24de4974a97c705f5353399a
I20260925 02:31:27.428598 12600 ts_tablet_manager.cc:1939] T 15c596ca24de4974a97c705f5353399a P ed20eea09c914726ace4c3fee97ac351: Deleting consensus metadata
I20260925 02:31:27.429090 12030 catalog_manager.cc:4841] TS ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807): tablet 15c596ca24de4974a97c705f5353399a (table test-table [id=4a8331d04a4d482d941a0e45613ad253]) successfully deleted
I20260925 02:31:27.429589 12601 ts_tablet_manager.cc:1918] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:27.429666 12601 log.cc:1196] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/15c596ca24de4974a97c705f5353399a
I20260925 02:31:27.429941 12601 ts_tablet_manager.cc:1939] T 15c596ca24de4974a97c705f5353399a P 3c4f448843e34f68aee74193e07520b3: Deleting consensus metadata
I20260925 02:31:27.430400 12029 catalog_manager.cc:4841] TS 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025): tablet 15c596ca24de4974a97c705f5353399a (table test-table [id=4a8331d04a4d482d941a0e45613ad253]) successfully deleted
I20260925 02:31:27.430449 12555 tablet_bootstrap.cc:492] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: No bootstrap required, opened a new log
I20260925 02:31:27.430502 12555 ts_tablet_manager.cc:1397] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 02:31:27.430665 12555 raft_consensus.cc:348] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.430747 12555 raft_consensus.cc:374] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.430769 12555 raft_consensus.cc:729] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5ae2e73e104faab3318a8d67cf2420, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.430814 12555 consensus_queue.cc:260] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.430915 12555 ts_tablet_manager.cc:1428] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:27.431349 12557 tablet_bootstrap.cc:492] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: No bootstrap required, opened a new log
I20260925 02:31:27.431398 12557 ts_tablet_manager.cc:1397] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 02:31:27.431553 12557 raft_consensus.cc:348] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.431627 12557 raft_consensus.cc:374] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.431648 12557 raft_consensus.cc:729] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed20eea09c914726ace4c3fee97ac351, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.431685 12557 consensus_queue.cc:260] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.431779 12557 ts_tablet_manager.cc:1428] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:27.433305 12556 tablet_bootstrap.cc:492] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: No bootstrap required, opened a new log
I20260925 02:31:27.433357 12556 ts_tablet_manager.cc:1397] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260925 02:31:27.433503 12556 raft_consensus.cc:348] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.433575 12556 raft_consensus.cc:374] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:27.433595 12556 raft_consensus.cc:729] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f448843e34f68aee74193e07520b3, State: Initialized, Role: FOLLOWER
I20260925 02:31:27.433637 12556 consensus_queue.cc:260] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.433733 12556 ts_tablet_manager.cc:1428] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:27.500128 12577 raft_consensus.cc:482] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:27.500243 12577 raft_consensus.cc:504] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.500409 12577 leader_election.cc:288] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807), 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645)
I20260925 02:31:27.500635 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed21d8ee0045259882b3b85d2450e7" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20eea09c914726ace4c3fee97ac351" is_pre_election: true
I20260925 02:31:27.500737 12204 raft_consensus.cc:2454] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 0.
I20260925 02:31:27.500790 12326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed21d8ee0045259882b3b85d2450e7" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ae2e73e104faab3318a8d67cf2420" is_pre_election: true
I20260925 02:31:27.500921 12326 raft_consensus.cc:2454] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 0.
I20260925 02:31:27.500934 12417 leader_election.cc:302] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f448843e34f68aee74193e07520b3, ed20eea09c914726ace4c3fee97ac351; no voters:
I20260925 02:31:27.501021 12577 raft_consensus.cc:2781] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:27.501049 12577 raft_consensus.cc:482] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:27.501070 12577 raft_consensus.cc:3037] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.502002 12577 raft_consensus.cc:504] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.502146 12577 leader_election.cc:288] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 election: Requested vote from peers ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807), 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645)
I20260925 02:31:27.502296 12326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed21d8ee0045259882b3b85d2450e7" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ae2e73e104faab3318a8d67cf2420"
I20260925 02:31:27.502367 12326 raft_consensus.cc:3037] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.502358 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed21d8ee0045259882b3b85d2450e7" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20eea09c914726ace4c3fee97ac351"
I20260925 02:31:27.502429 12204 raft_consensus.cc:3037] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:27.503178 12326 raft_consensus.cc:2454] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 1.
I20260925 02:31:27.503232 12204 raft_consensus.cc:2454] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 1.
I20260925 02:31:27.503358 12419 leader_election.cc:302] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f448843e34f68aee74193e07520b3, 8e5ae2e73e104faab3318a8d67cf2420; no voters:
I20260925 02:31:27.503659 12577 raft_consensus.cc:2781] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:27.503705 12577 raft_consensus.cc:686] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4f448843e34f68aee74193e07520b3, State: Running, Role: LEADER
I20260925 02:31:27.503747 12577 consensus_queue.cc:237] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:27.504382 12045 catalog_manager.cc:5495] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4f448843e34f68aee74193e07520b3 (127.10.13.3). New cstate: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } health_report { overall_health: HEALTHY } } }
I20260925 02:31:27.524365 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.521398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.10.13.3:37025)
0925 02:31:27.521450 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.521452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.521462 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:27.521536 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.521536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.521547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.521607 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:27.521749 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.522543 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.523619 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.523897 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.523975 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.524118 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.524154 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.524208 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.524267 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.524287 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:31:27.529464 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.526681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:37025 (local address 127.10.13.3:47257)
0925 02:31:27.526724 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:27.526734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:27.526766 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:27.527036 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:27.527037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:27.527038 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:27.527070 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.527071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.527889 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.527890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.528700 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:27.528703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.529157 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.529158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:27.529194 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.529197 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.529324 (+ 127us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.529325 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 02:31:27.529356 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 02:31:27.529395 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 02:31:27.529742 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:27.526887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:47257 (local address 127.10.13.3:37025)
0925 02:31:27.526936 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:27.526938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:27.526943 (+ 5us) server_negotiation.cc:407] Connection header received
0925 02:31:27.526959 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:27.526959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:27.526967 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:27.527006 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:27.527125 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.527801 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.528803 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:27.529077 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:27.529132 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:27.529252 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:27.529278 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:27.529310 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:27.529680 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 02:31:27.529696 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:31:27.531951 12326 raft_consensus.cc:1261] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Refusing update from remote peer 3c4f448843e34f68aee74193e07520b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:27.532002 12204 raft_consensus.cc:1261] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Refusing update from remote peer 3c4f448843e34f68aee74193e07520b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:27.532142 12606 consensus_queue.cc:1035] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:27.532349 12606 consensus_queue.cc:1035] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:31:27.537870 12582 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:31:29.060531 12043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:29.060639 12043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:29.061661 12043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: Sending DeleteTablet for 3 replicas of tablet e2ed21d8ee0045259882b3b85d2450e7
I20260925 02:31:29.061964 12325 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ed21d8ee0045259882b3b85d2450e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:29 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39898
I20260925 02:31:29.062002 12465 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ed21d8ee0045259882b3b85d2450e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:29 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:59400
I20260925 02:31:29.062026 12183 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ed21d8ee0045259882b3b85d2450e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:29 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:37544
I20260925 02:31:29.062240 12631 tablet_replica.cc:330] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: stopping tablet replica
I20260925 02:31:29.062302 12631 raft_consensus.cc:2229] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:29.062400 12631 raft_consensus.cc:2258] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:29.062717 12631 ts_tablet_manager.cc:1905] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:29.064742 12631 ts_tablet_manager.cc:1918] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:31:29.064829 12631 log.cc:1196] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/e2ed21d8ee0045259882b3b85d2450e7
I20260925 02:31:29.065075 12630 tablet_replica.cc:330] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: stopping tablet replica
I20260925 02:31:29.065136 12631 ts_tablet_manager.cc:1939] T e2ed21d8ee0045259882b3b85d2450e7 P 3c4f448843e34f68aee74193e07520b3: Deleting consensus metadata
I20260925 02:31:29.065146 12630 raft_consensus.cc:2229] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:29.065191 12630 raft_consensus.cc:2258] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:29.065405 12629 tablet_replica.cc:330] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: stopping tablet replica
I20260925 02:31:29.065445 12630 ts_tablet_manager.cc:1905] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:29.065490 12629 raft_consensus.cc:2229] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:29.065550 12629 raft_consensus.cc:2258] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:29.065768 12629 ts_tablet_manager.cc:1905] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:29.065927 12029 catalog_manager.cc:4841] TS 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025): tablet e2ed21d8ee0045259882b3b85d2450e7 (table test-table [id=525c673a52a24905ada9020644c05b0e]) successfully deleted
I20260925 02:31:29.067381 12629 ts_tablet_manager.cc:1918] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:31:29.067410 12630 ts_tablet_manager.cc:1918] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:31:29.067461 12629 log.cc:1196] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/e2ed21d8ee0045259882b3b85d2450e7
I20260925 02:31:29.067478 12630 log.cc:1196] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/e2ed21d8ee0045259882b3b85d2450e7
I20260925 02:31:29.067715 12630 ts_tablet_manager.cc:1939] T e2ed21d8ee0045259882b3b85d2450e7 P ed20eea09c914726ace4c3fee97ac351: Deleting consensus metadata
I20260925 02:31:29.067732 12629 ts_tablet_manager.cc:1939] T e2ed21d8ee0045259882b3b85d2450e7 P 8e5ae2e73e104faab3318a8d67cf2420: Deleting consensus metadata
I20260925 02:31:29.068410 12030 catalog_manager.cc:4841] TS ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807): tablet e2ed21d8ee0045259882b3b85d2450e7 (table test-table [id=525c673a52a24905ada9020644c05b0e]) successfully deleted
I20260925 02:31:29.068428 12032 catalog_manager.cc:4841] TS 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645): tablet e2ed21d8ee0045259882b3b85d2450e7 (table test-table [id=525c673a52a24905ada9020644c05b0e]) successfully deleted
Sep 25 02:31:29 dist-test-slave-q8mg krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:31:29 dist-test-slave-q8mg krb5kdc[12000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:29.100382 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.079587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.10.13.62:41731)
0925 02:31:29.079785 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:29.079787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:29.079797 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:29.079887 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:29.079888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:29.079898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:29.079959 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:29.080526 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.081195 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.081993 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.082352 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.082429 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.096092 (+ 13663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:29.096096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:29.096097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:29.096108 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:29.098673 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:29.099133 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:29.099135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:29.099135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:29.099214 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:29.099454 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:29.099456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:29.099456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:29.099632 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:29.099978 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:29.100152 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 02:31:29.100253 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":134,"threads_started":1}
I20260925 02:31:29.102834 12043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:29.103125 12043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:29.104818 12325 tablet_service.cc:1467] Processing CreateTablet for tablet 3d58d6416e1a4e499c7294447eb0771a (DEFAULT_TABLE table=test-table [id=a165d37e531c4b11b26e337f780edde7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:29.104915 12465 tablet_service.cc:1467] Processing CreateTablet for tablet 3d58d6416e1a4e499c7294447eb0771a (DEFAULT_TABLE table=test-table [id=a165d37e531c4b11b26e337f780edde7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:29.104990 12325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d58d6416e1a4e499c7294447eb0771a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:29.105075 12465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d58d6416e1a4e499c7294447eb0771a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:29.105098 12183 tablet_service.cc:1467] Processing CreateTablet for tablet 3d58d6416e1a4e499c7294447eb0771a (DEFAULT_TABLE table=test-table [id=a165d37e531c4b11b26e337f780edde7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:29.105266 12183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d58d6416e1a4e499c7294447eb0771a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:29.106832 12640 tablet_bootstrap.cc:492] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: Bootstrap starting.
I20260925 02:31:29.107224 12640 tablet_bootstrap.cc:654] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:29.107437 12642 tablet_bootstrap.cc:492] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: Bootstrap starting.
I20260925 02:31:29.107466 12641 tablet_bootstrap.cc:492] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: Bootstrap starting.
I20260925 02:31:29.107810 12641 tablet_bootstrap.cc:654] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:29.107968 12642 tablet_bootstrap.cc:654] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:29.108490 12641 tablet_bootstrap.cc:492] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: No bootstrap required, opened a new log
I20260925 02:31:29.108572 12641 ts_tablet_manager.cc:1397] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:29.108655 12640 tablet_bootstrap.cc:492] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: No bootstrap required, opened a new log
I20260925 02:31:29.108697 12641 raft_consensus.cc:348] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.108731 12640 ts_tablet_manager.cc:1397] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:29.108775 12641 raft_consensus.cc:374] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:29.108803 12641 raft_consensus.cc:729] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5ae2e73e104faab3318a8d67cf2420, State: Initialized, Role: FOLLOWER
I20260925 02:31:29.108899 12641 consensus_queue.cc:260] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.109105 12641 ts_tablet_manager.cc:1428] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:29.109375 12642 tablet_bootstrap.cc:492] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: No bootstrap required, opened a new log
I20260925 02:31:29.109341 12640 raft_consensus.cc:348] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.109464 12640 raft_consensus.cc:374] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:29.109500 12640 raft_consensus.cc:729] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f448843e34f68aee74193e07520b3, State: Initialized, Role: FOLLOWER
I20260925 02:31:29.109580 12642 ts_tablet_manager.cc:1397] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:29.109589 12640 consensus_queue.cc:260] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.109763 12640 ts_tablet_manager.cc:1428] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:29.109876 12642 raft_consensus.cc:348] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.110045 12642 raft_consensus.cc:374] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:29.110118 12642 raft_consensus.cc:729] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed20eea09c914726ace4c3fee97ac351, State: Initialized, Role: FOLLOWER
I20260925 02:31:29.110206 12642 consensus_queue.cc:260] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.110381 12642 ts_tablet_manager.cc:1428] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:31:29.215130 12563 raft_consensus.cc:482] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:29.215287 12563 raft_consensus.cc:504] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.215519 12563 leader_election.cc:288] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807), 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645)
I20260925 02:31:29.215788 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d58d6416e1a4e499c7294447eb0771a" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20eea09c914726ace4c3fee97ac351" is_pre_election: true
I20260925 02:31:29.215809 12345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d58d6416e1a4e499c7294447eb0771a" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ae2e73e104faab3318a8d67cf2420" is_pre_election: true
I20260925 02:31:29.215915 12204 raft_consensus.cc:2454] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 0.
I20260925 02:31:29.216023 12345 raft_consensus.cc:2454] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 0.
I20260925 02:31:29.216104 12417 leader_election.cc:302] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f448843e34f68aee74193e07520b3, ed20eea09c914726ace4c3fee97ac351; no voters:
I20260925 02:31:29.216189 12563 raft_consensus.cc:2781] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:29.216245 12563 raft_consensus.cc:482] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:29.216274 12563 raft_consensus.cc:3037] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:29.217154 12563 raft_consensus.cc:504] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.217310 12563 leader_election.cc:288] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 election: Requested vote from peers ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807), 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645)
I20260925 02:31:29.217474 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d58d6416e1a4e499c7294447eb0771a" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed20eea09c914726ace4c3fee97ac351"
I20260925 02:31:29.217554 12204 raft_consensus.cc:3037] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:29.217520 12345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d58d6416e1a4e499c7294447eb0771a" candidate_uuid: "3c4f448843e34f68aee74193e07520b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ae2e73e104faab3318a8d67cf2420"
I20260925 02:31:29.217626 12345 raft_consensus.cc:3037] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:29.218150 12204 raft_consensus.cc:2454] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 1.
I20260925 02:31:29.218319 12417 leader_election.cc:302] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f448843e34f68aee74193e07520b3, ed20eea09c914726ace4c3fee97ac351; no voters:
I20260925 02:31:29.218422 12563 raft_consensus.cc:2781] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:29.218449 12345 raft_consensus.cc:2454] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4f448843e34f68aee74193e07520b3 in term 1.
I20260925 02:31:29.218497 12563 raft_consensus.cc:686] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4f448843e34f68aee74193e07520b3, State: Running, Role: LEADER
I20260925 02:31:29.218551 12563 consensus_queue.cc:237] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.219251 12045 catalog_manager.cc:5495] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4f448843e34f68aee74193e07520b3 (127.10.13.3). New cstate: current_term: 1 leader_uuid: "3c4f448843e34f68aee74193e07520b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } health_report { overall_health: HEALTHY } } }
I20260925 02:31:29.227829 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.225697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.10.13.3:37025)
0925 02:31:29.225848 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:29.225849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:29.225857 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:29.225886 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:29.225886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:29.225897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:29.225957 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:29.226072 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.226576 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.227306 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.227485 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.227534 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.227653 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:29.227686 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:29.227720 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:29.227752 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 02:31:29.227768 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":105,"threads_started":1}
I20260925 02:31:29.230122 12204 raft_consensus.cc:1261] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Refusing update from remote peer 3c4f448843e34f68aee74193e07520b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:29.230125 12345 raft_consensus.cc:1261] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Refusing update from remote peer 3c4f448843e34f68aee74193e07520b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:29.230334 12563 consensus_queue.cc:1035] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:29.230497 12617 consensus_queue.cc:1035] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:29.240350 12044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:29.240461 12044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:29.241145 12044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: Sending DeleteTablet for 3 replicas of tablet 3d58d6416e1a4e499c7294447eb0771a
I20260925 02:31:29.241417 12183 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d58d6416e1a4e499c7294447eb0771a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:29 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:37544
I20260925 02:31:29.241451 12325 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d58d6416e1a4e499c7294447eb0771a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:29 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39898
I20260925 02:31:29.241468 12465 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d58d6416e1a4e499c7294447eb0771a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:29 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:59400
I20260925 02:31:29.241535 12629 tablet_replica.cc:330] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: stopping tablet replica
I20260925 02:31:29.241569 12630 tablet_replica.cc:330] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: stopping tablet replica
I20260925 02:31:29.241590 12629 raft_consensus.cc:2229] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:29.241633 12629 raft_consensus.cc:2258] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:29.241636 12630 raft_consensus.cc:2229] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:29.241677 12630 raft_consensus.cc:2258] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:29.241750 12631 tablet_replica.cc:330] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: stopping tablet replica
I20260925 02:31:29.241806 12631 raft_consensus.cc:2229] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:29.241864 12631 raft_consensus.cc:2258] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:29.241968 12629 ts_tablet_manager.cc:1905] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:29.242137 12630 ts_tablet_manager.cc:1905] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:29.242306 12631 ts_tablet_manager.cc:1905] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:29.243772 12630 ts_tablet_manager.cc:1918] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:29.243772 12629 ts_tablet_manager.cc:1918] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:29.243871 12629 log.cc:1196] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/3d58d6416e1a4e499c7294447eb0771a
I20260925 02:31:29.243880 12630 log.cc:1196] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/3d58d6416e1a4e499c7294447eb0771a
I20260925 02:31:29.243882 12631 ts_tablet_manager.cc:1918] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:29.243929 12631 log.cc:1196] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/3d58d6416e1a4e499c7294447eb0771a
I20260925 02:31:29.244115 12629 ts_tablet_manager.cc:1939] T 3d58d6416e1a4e499c7294447eb0771a P 8e5ae2e73e104faab3318a8d67cf2420: Deleting consensus metadata
I20260925 02:31:29.244160 12631 ts_tablet_manager.cc:1939] T 3d58d6416e1a4e499c7294447eb0771a P 3c4f448843e34f68aee74193e07520b3: Deleting consensus metadata
I20260925 02:31:29.244235 12630 ts_tablet_manager.cc:1939] T 3d58d6416e1a4e499c7294447eb0771a P ed20eea09c914726ace4c3fee97ac351: Deleting consensus metadata
I20260925 02:31:29.244451 12032 catalog_manager.cc:4841] TS 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645): tablet 3d58d6416e1a4e499c7294447eb0771a (table test-table [id=a165d37e531c4b11b26e337f780edde7]) successfully deleted
I20260925 02:31:29.244393 12044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:31:29.244561 12030 catalog_manager.cc:4841] TS ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807): tablet 3d58d6416e1a4e499c7294447eb0771a (table test-table [id=a165d37e531c4b11b26e337f780edde7]) successfully deleted
W20260925 02:31:29.244560 12044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:29.244747 12029 catalog_manager.cc:4841] TS 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025): tablet 3d58d6416e1a4e499c7294447eb0771a (table test-table [id=a165d37e531c4b11b26e337f780edde7]) successfully deleted
I20260925 02:31:29.245890 12183 tablet_service.cc:1467] Processing CreateTablet for tablet 17120de386a64d03adfcbfacf7ab8c9e (DEFAULT_TABLE table=test-table [id=d4cf199dd67b411f9a8c46caabd5c602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:29.245913 12325 tablet_service.cc:1467] Processing CreateTablet for tablet 17120de386a64d03adfcbfacf7ab8c9e (DEFAULT_TABLE table=test-table [id=d4cf199dd67b411f9a8c46caabd5c602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:29.245960 12465 tablet_service.cc:1467] Processing CreateTablet for tablet 17120de386a64d03adfcbfacf7ab8c9e (DEFAULT_TABLE table=test-table [id=d4cf199dd67b411f9a8c46caabd5c602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:29.246021 12325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17120de386a64d03adfcbfacf7ab8c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:29.246021 12183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17120de386a64d03adfcbfacf7ab8c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:29.246054 12465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17120de386a64d03adfcbfacf7ab8c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:29.246941 12642 tablet_bootstrap.cc:492] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351: Bootstrap starting.
I20260925 02:31:29.247184 12641 tablet_bootstrap.cc:492] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: Bootstrap starting.
I20260925 02:31:29.247202 12640 tablet_bootstrap.cc:492] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: Bootstrap starting.
I20260925 02:31:29.247457 12641 tablet_bootstrap.cc:654] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:29.247584 12640 tablet_bootstrap.cc:654] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:29.247584 12642 tablet_bootstrap.cc:654] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:29.247978 12641 tablet_bootstrap.cc:492] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: No bootstrap required, opened a new log
I20260925 02:31:29.248026 12641 ts_tablet_manager.cc:1397] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:29.248139 12641 raft_consensus.cc:348] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.248198 12641 raft_consensus.cc:374] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:29.248212 12641 raft_consensus.cc:729] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5ae2e73e104faab3318a8d67cf2420, State: Initialized, Role: FOLLOWER
I20260925 02:31:29.248200 12640 tablet_bootstrap.cc:492] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: No bootstrap required, opened a new log
I20260925 02:31:29.248255 12640 ts_tablet_manager.cc:1397] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:29.248278 12641 consensus_queue.cc:260] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.248383 12641 ts_tablet_manager.cc:1428] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:29.248384 12640 raft_consensus.cc:348] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.248490 12640 raft_consensus.cc:374] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:29.248519 12640 raft_consensus.cc:729] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f448843e34f68aee74193e07520b3, State: Initialized, Role: FOLLOWER
I20260925 02:31:29.248566 12640 consensus_queue.cc:260] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.248672 12640 ts_tablet_manager.cc:1428] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:29.249135 12642 tablet_bootstrap.cc:492] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351: No bootstrap required, opened a new log
I20260925 02:31:29.249189 12642 ts_tablet_manager.cc:1397] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:29.249356 12642 raft_consensus.cc:348] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.249446 12642 raft_consensus.cc:374] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:29.249469 12642 raft_consensus.cc:729] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed20eea09c914726ace4c3fee97ac351, State: Initialized, Role: FOLLOWER
I20260925 02:31:29.249539 12642 consensus_queue.cc:260] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.249635 12642 ts_tablet_manager.cc:1428] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:29.317070 12647 raft_consensus.cc:482] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:29.317183 12647 raft_consensus.cc:504] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.317636 12647 leader_election.cc:288] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645), 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025)
I20260925 02:31:29.320904 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.317807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:40645 (local address 127.10.13.1:57363)
0925 02:31:29.317981 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:29.317990 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:29.318036 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:29.318314 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:29.318315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:29.318318 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:29.318421 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:29.318423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.319129 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.319130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:29.319942 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:29.319944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.320553 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.320554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:29.320650 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.320655 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:29.320706 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:31:29.320750 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":29,"thread_start_us":85,"threads_started":1}
I20260925 02:31:29.320937 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.318009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:32951 (local address 127.10.13.3:37025)
0925 02:31:29.318070 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:29.318072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:29.318082 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:29.318115 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:29.318116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:29.318127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:29.318233 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:29.318415 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.318949 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.320111 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.320457 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.320545 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.320557 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:29.320806 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 02:31:29.320849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:31:29.321117 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.318004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:57363 (local address 127.10.13.2:40645)
0925 02:31:29.318153 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:29.318155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:29.318162 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:31:29.318202 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:29.318203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:29.318215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:29.318287 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:29.318520 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.319022 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.320039 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.320456 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.320544 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.320557 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:29.321009 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 02:31:29.321059 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260925 02:31:29.321334 12345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17120de386a64d03adfcbfacf7ab8c9e" candidate_uuid: "ed20eea09c914726ace4c3fee97ac351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ae2e73e104faab3318a8d67cf2420" is_pre_election: true
I20260925 02:31:29.321437 12345 raft_consensus.cc:2454] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed20eea09c914726ace4c3fee97ac351 in term 0.
I20260925 02:31:29.321503 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.317807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:37025 (local address 127.10.13.1:32951)
0925 02:31:29.317942 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:29.317957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:29.318036 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:29.318244 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:29.318246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:29.318248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:29.318334 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:29.318337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.319030 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.319031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:29.319936 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:29.319938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.320653 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.320654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:29.320734 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.320739 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:29.320795 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:31:29.320836 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
I20260925 02:31:29.321627 12137 leader_election.cc:302] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e5ae2e73e104faab3318a8d67cf2420, ed20eea09c914726ace4c3fee97ac351; no voters:
I20260925 02:31:29.321727 12647 raft_consensus.cc:2781] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:29.321769 12647 raft_consensus.cc:482] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:29.321789 12647 raft_consensus.cc:3037] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:29.321779 12486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17120de386a64d03adfcbfacf7ab8c9e" candidate_uuid: "ed20eea09c914726ace4c3fee97ac351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4f448843e34f68aee74193e07520b3" is_pre_election: true
I20260925 02:31:29.321877 12486 raft_consensus.cc:2454] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed20eea09c914726ace4c3fee97ac351 in term 0.
I20260925 02:31:29.322515 12647 raft_consensus.cc:504] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.322638 12647 leader_election.cc:288] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [CANDIDATE]: Term 1 election: Requested vote from peers 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645), 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025)
I20260925 02:31:29.322820 12345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17120de386a64d03adfcbfacf7ab8c9e" candidate_uuid: "ed20eea09c914726ace4c3fee97ac351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ae2e73e104faab3318a8d67cf2420"
I20260925 02:31:29.322887 12345 raft_consensus.cc:3037] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:29.322822 12486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17120de386a64d03adfcbfacf7ab8c9e" candidate_uuid: "ed20eea09c914726ace4c3fee97ac351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4f448843e34f68aee74193e07520b3"
I20260925 02:31:29.322930 12486 raft_consensus.cc:3037] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:29.323540 12345 raft_consensus.cc:2454] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed20eea09c914726ace4c3fee97ac351 in term 1.
I20260925 02:31:29.323719 12137 leader_election.cc:302] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e5ae2e73e104faab3318a8d67cf2420, ed20eea09c914726ace4c3fee97ac351; no voters:
I20260925 02:31:29.323773 12486 raft_consensus.cc:2454] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed20eea09c914726ace4c3fee97ac351 in term 1.
I20260925 02:31:29.323838 12647 raft_consensus.cc:2781] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:29.324010 12647 raft_consensus.cc:686] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [term 1 LEADER]: Becoming Leader. State: Replica: ed20eea09c914726ace4c3fee97ac351, State: Running, Role: LEADER
I20260925 02:31:29.324098 12647 consensus_queue.cc:237] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } }
I20260925 02:31:29.324817 12044 catalog_manager.cc:5495] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed20eea09c914726ace4c3fee97ac351 (127.10.13.1). New cstate: current_term: 1 leader_uuid: "ed20eea09c914726ace4c3fee97ac351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed20eea09c914726ace4c3fee97ac351" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 44807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20260925 02:31:29.345854 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.343133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.10.13.1:44807)
0925 02:31:29.343335 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:29.343337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:29.343350 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:31:29.343388 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:29.343389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:29.343410 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:29.343503 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:29.343647 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.344158 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.344937 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.345187 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.345268 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.345421 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:29.345548 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:29.345665 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:29.345710 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 02:31:29.345730 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":117,"threads_started":1}
I20260925 02:31:29.351250 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.348619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:54471 (local address 127.10.13.3:37025)
0925 02:31:29.348689 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:29.348692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:29.348701 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:29.348730 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:29.348731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:29.348744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:29.348823 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:29.349012 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.349744 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.350550 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.350798 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.350854 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.351022 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:29.351059 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:29.351101 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:29.351171 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:31:29.351191 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:31:29.351279 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.348355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:37025 (local address 127.10.13.1:54471)
0925 02:31:29.348545 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:29.348559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:29.348614 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:29.348836 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:29.348837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:29.348839 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:29.348935 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:29.348937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.349817 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.349818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:29.350460 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:29.350462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.350893 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.350894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:29.350939 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.350954 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:29.351133 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:29.351134 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 02:31:29.351169 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 02:31:29.351204 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260925 02:31:29.353513 12486 raft_consensus.cc:1261] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Refusing update from remote peer ed20eea09c914726ace4c3fee97ac351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:29.353523 12345 raft_consensus.cc:1261] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Refusing update from remote peer ed20eea09c914726ace4c3fee97ac351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:29.353801 12647 consensus_queue.cc:1035] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5ae2e73e104faab3318a8d67cf2420" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:29.354012 12660 consensus_queue.cc:1035] T 17120de386a64d03adfcbfacf7ab8c9e P ed20eea09c914726ace4c3fee97ac351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4f448843e34f68aee74193e07520b3" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:29.361776 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.359248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:57337 (local address 127.10.13.1:44807)
0925 02:31:29.359296 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:29.359299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:29.359361 (+ 62us) server_negotiation.cc:407] Connection header received
0925 02:31:29.359427 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:29.359428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:29.359440 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:29.359488 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:29.359648 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.360144 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.360919 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.361243 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.361325 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.361497 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:29.361540 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:29.361615 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:29.361679 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:31:29.361701 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:31:29.361802 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:29.359138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:44807 (local address 127.10.13.3:57337)
0925 02:31:29.359308 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:29.359319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:29.359376 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:29.359534 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:29.359540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:29.359542 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:29.359597 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:29.359598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.360225 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.360226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:29.360800 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:29.360802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:29.361338 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:29.361339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:29.361417 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:29.361425 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:29.361620 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:29.361622 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:31:29.361674 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:31:29.361725 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":121,"threads_started":1}
W20260925 02:31:29.363343 12615 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:31:30.887168 12043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:30.887271 12043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:30.888170 12043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bae0b765c034241bdf9f78b4e1dd531: Sending DeleteTablet for 3 replicas of tablet 17120de386a64d03adfcbfacf7ab8c9e
I20260925 02:31:30.888446 12325 tablet_service.cc:1514] Processing DeleteTablet for tablet 17120de386a64d03adfcbfacf7ab8c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:30 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39898
I20260925 02:31:30.888446 12465 tablet_service.cc:1514] Processing DeleteTablet for tablet 17120de386a64d03adfcbfacf7ab8c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:30 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:59400
I20260925 02:31:30.888453 12183 tablet_service.cc:1514] Processing DeleteTablet for tablet 17120de386a64d03adfcbfacf7ab8c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:30 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:37544
I20260925 02:31:30.888589 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 12118
I20260925 02:31:30.889638 12697 tablet_replica.cc:330] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: stopping tablet replica
I20260925 02:31:30.889631 12695 tablet_replica.cc:330] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: stopping tablet replica
I20260925 02:31:30.889704 12697 raft_consensus.cc:2229] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:30.889704 12695 raft_consensus.cc:2229] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:30.889742 12697 raft_consensus.cc:2258] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:30.889742 12695 raft_consensus.cc:2258] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:30.889987 12697 ts_tablet_manager.cc:1905] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:30.890097 12695 ts_tablet_manager.cc:1905] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:30.892120 12697 ts_tablet_manager.cc:1918] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:31:30.892120 12695 ts_tablet_manager.cc:1918] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:31:30.892186 12697 log.cc:1196] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/17120de386a64d03adfcbfacf7ab8c9e
I20260925 02:31:30.892186 12695 log.cc:1196] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/17120de386a64d03adfcbfacf7ab8c9e
I20260925 02:31:30.892426 12697 ts_tablet_manager.cc:1939] T 17120de386a64d03adfcbfacf7ab8c9e P 3c4f448843e34f68aee74193e07520b3: Deleting consensus metadata
I20260925 02:31:30.892426 12695 ts_tablet_manager.cc:1939] T 17120de386a64d03adfcbfacf7ab8c9e P 8e5ae2e73e104faab3318a8d67cf2420: Deleting consensus metadata
I20260925 02:31:30.892792 12032 catalog_manager.cc:4841] TS 8e5ae2e73e104faab3318a8d67cf2420 (127.10.13.2:40645): tablet 17120de386a64d03adfcbfacf7ab8c9e (table test-table [id=d4cf199dd67b411f9a8c46caabd5c602]) successfully deleted
I20260925 02:31:30.892937 12029 catalog_manager.cc:4841] TS 3c4f448843e34f68aee74193e07520b3 (127.10.13.3:37025): tablet 17120de386a64d03adfcbfacf7ab8c9e (table test-table [id=d4cf199dd67b411f9a8c46caabd5c602]) successfully deleted
W20260925 02:31:30.896999 12030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:44807 (error 108)
I20260925 02:31:30.897202 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 12259
I20260925 02:31:30.899199 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:30.898880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:44807 (local address 127.0.0.1:37550)
0925 02:31:30.899110 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:30.899161 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
W20260925 02:31:30.899523 12030 catalog_manager.cc:4584] TS ed20eea09c914726ace4c3fee97ac351 (127.10.13.1:44807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17120de386a64d03adfcbfacf7ab8c9e: Network error: Client connection negotiation failed: client connection to 127.10.13.1:44807: connect: Connection refused (error 111)
I20260925 02:31:30.903909 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 12400
I20260925 02:31:30.910427 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 12013
2026-09-25T02:31:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5628 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:30 dist-test-slave-q8mg krb5kdc[12702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:30 dist-test-slave-q8mg krb5kdc[12702](info): set up 2 sockets
Sep 25 02:31:30 dist-test-slave-q8mg krb5kdc[12702](info): commencing operation
krb5kdc: starting...
W20260925 02:31:31.955331 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:32.078162 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:44981
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:44981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:32.162668 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:32.162941 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:32.162968 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:32.164170 12715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:32.164214 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:32.164228 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:32.164242 12715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:32.164253 12715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:32.165756 12715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:44981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:44981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:32.165977 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:32.166164 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:32.171201 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:32.171201 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:32.171258 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:32.171517 12715 server_base.cc:1034] running on GCE node
I20260925 02:31:32.171777 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:32.172082 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:32.173285 12715 hybrid_clock.cc:648] HybridClock initialized: now 1790303492173277 us; error 56 us; skew 500 ppm
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:32.175812 12715 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:32.176332 12715 webserver.cc:460] Webserver started at http://127.10.13.62:33095/ using document root <none> and password file <none>
I20260925 02:31:32.176599 12715 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:32.176672 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:32.176824 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:32.178021 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "d7b973cbccad4856a4b8c8eaa3da9747"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "768cb70a785dd6a5f3dd82774fd811c7"
server_key_iv: "1231320b4b55a2ad5e1647bd0ecb6979"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.178406 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "d7b973cbccad4856a4b8c8eaa3da9747"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "768cb70a785dd6a5f3dd82774fd811c7"
server_key_iv: "1231320b4b55a2ad5e1647bd0ecb6979"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.179806 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:32.180593 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.180784 12715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:32.180919 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "d7b973cbccad4856a4b8c8eaa3da9747"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "768cb70a785dd6a5f3dd82774fd811c7"
server_key_iv: "1231320b4b55a2ad5e1647bd0ecb6979"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.180998 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:32.199596 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:32.201683 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:32.201865 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:32.206223 12715 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:44981
I20260925 02:31:32.206265 12783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:44981 every 8 connection(s)
I20260925 02:31:32.206645 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:32.207609 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:32.210085 12784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747: Bootstrap starting.
I20260925 02:31:32.210738 12784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:32.211053 12784 log.cc:824] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:32.211800 12784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747: No bootstrap required, opened a new log
I20260925 02:31:32.213691 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 12715
I20260925 02:31:32.213601 12784 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b973cbccad4856a4b8c8eaa3da9747" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44981 } }
I20260925 02:31:32.213769 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:32.213771 12784 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:32.213812 12784 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b973cbccad4856a4b8c8eaa3da9747, State: Initialized, Role: FOLLOWER
I20260925 02:31:32.213963 10292 external_mini_cluster.cc:1428] Setting key 5ca69d205277fc8fd9f7a85d65f23bed
I20260925 02:31:32.214027 12784 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b973cbccad4856a4b8c8eaa3da9747" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44981 } }
I20260925 02:31:32.214129 12784 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:32.214179 12784 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:32.214234 12784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:32.215305 12784 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b973cbccad4856a4b8c8eaa3da9747" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44981 } }
I20260925 02:31:32.215444 12784 leader_election.cc:302] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7b973cbccad4856a4b8c8eaa3da9747; no voters:
I20260925 02:31:32.215670 12784 leader_election.cc:288] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:32.215734 12789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:32.215939 12789 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [term 1 LEADER]: Becoming Leader. State: Replica: d7b973cbccad4856a4b8c8eaa3da9747, State: Running, Role: LEADER
I20260925 02:31:32.215957 12784 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:32.216101 12789 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b973cbccad4856a4b8c8eaa3da9747" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44981 } }
I20260925 02:31:32.216713 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7b973cbccad4856a4b8c8eaa3da9747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b973cbccad4856a4b8c8eaa3da9747" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44981 } } }
I20260925 02:31:32.216799 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:32.217064 12791 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7b973cbccad4856a4b8c8eaa3da9747. Latest consensus state: current_term: 1 leader_uuid: "d7b973cbccad4856a4b8c8eaa3da9747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b973cbccad4856a4b8c8eaa3da9747" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44981 } } }
I20260925 02:31:32.217199 12791 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:32.217344 12798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:32.217867 12798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:32.219703 12798 catalog_manager.cc:1347] Generated new cluster ID: 72ec560aadec44f89c72ec801f4c873e
I20260925 02:31:32.219761 12798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:32.226547 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:32.214525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.10.13.62:44981)
0925 02:31:32.214734 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:32.214746 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:32.214780 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:31:32.214900 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:32.214906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:32.214951 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:32.215097 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:32.215864 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.217401 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.218365 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.218956 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.219099 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:32.221878 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:32.221882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:32.221883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:32.221896 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:32.224584 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:32.225005 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:32.225008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:32.225009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:32.225132 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:32.225305 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:32.225306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:32.225306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:32.225599 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:32.225738 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:32.226165 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 02:31:32.226306 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260925 02:31:32.236013 12798 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:32.236723 12798 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:32.243777 12798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7b973cbccad4856a4b8c8eaa3da9747: Generated new TSK 0
I20260925 02:31:32.243985 12798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:32.277042 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44981
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:32.367969 12810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:32.368170 12810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:32.368194 12810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:32.369439 12810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:32.369481 12810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:32.369508 12810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:32.370913 12810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:32.371117 12810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:32.371299 12810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:32.376583 12818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:32.376583 12816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:32.376578 12815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:32.377171 12810 server_base.cc:1034] running on GCE node
I20260925 02:31:32.377395 12810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:32.377626 12810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:32.378777 12810 hybrid_clock.cc:648] HybridClock initialized: now 1790303492378763 us; error 29 us; skew 500 ppm
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:32.381361 12810 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:32.381841 12810 webserver.cc:460] Webserver started at http://127.10.13.1:40257/ using document root <none> and password file <none>
I20260925 02:31:32.382107 12810 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:32.382176 12810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:32.382333 12810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:32.383391 12810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "a5ae2525dc7149a89c9d114e0f22d2bc"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "31f6214ee31a0023fb756767d208c7e6"
server_key_iv: "1a9d5ffa3a9ad5f53020a6b6dfa7ddd1"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.383873 12810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "a5ae2525dc7149a89c9d114e0f22d2bc"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "31f6214ee31a0023fb756767d208c7e6"
server_key_iv: "1a9d5ffa3a9ad5f53020a6b6dfa7ddd1"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.385491 12810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:32.386313 12825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.386502 12810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:32.386618 12810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "a5ae2525dc7149a89c9d114e0f22d2bc"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "31f6214ee31a0023fb756767d208c7e6"
server_key_iv: "1a9d5ffa3a9ad5f53020a6b6dfa7ddd1"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.386701 12810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:32.416721 12810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:32.418855 12810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:32.419028 12810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:32.419288 12810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:32.419621 12810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:32.419674 12810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.419715 12810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:32.419744 12810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.425884 12810 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:42463
I20260925 02:31:32.425947 12939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:42463 every 8 connection(s)
I20260925 02:31:32.426340 12810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:32.432789 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 12810
I20260925 02:31:32.432916 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:32.433125 10292 external_mini_cluster.cc:1428] Setting key 1bdc0b64c9302a09d15f4d4df822edcc
I20260925 02:31:32.436616 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:32.427220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:45097 (local address 127.10.13.62:44981)
0925 02:31:32.427292 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:32.427294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:32.427522 (+ 228us) server_negotiation.cc:407] Connection header received
0925 02:31:32.428075 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:32.428077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:32.428095 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:32.428151 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:32.429265 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.429783 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.430819 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.431032 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.431112 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:32.433510 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:32.433511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:32.433512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:32.433540 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:32.435100 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:32.435585 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:32.435587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:32.435587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:32.435650 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:32.435925 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:32.435926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:32.435926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:32.436046 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:32.436128 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:32.436418 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 02:31:32.436506 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:31:32.436820 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:32.427192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44981 (local address 127.10.13.1:45097)
0925 02:31:32.427388 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:32.427427 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:32.427960 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:32.428242 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:32.428245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:32.428840 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:32.429139 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:32.429145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.429903 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.429904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:32.430722 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:32.430724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.431123 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.431125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:32.431258 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:32.431778 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:32.431788 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:32.433397 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:32.435210 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:32.435219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:32.435221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:32.435497 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:32.435721 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:32.435722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:32.435723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:32.435846 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:32.436119 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:32.436127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:32.436280 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 02:31:32.436395 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:31:32.436564 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260925 02:31:32.437376 12940 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44981
I20260925 02:31:32.437515 12940 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:32.437716 12940 heartbeater.cc:511] Master 127.10.13.62:44981 requested a full tablet report, sending...
I20260925 02:31:32.438325 12747 ts_manager.cc:194] Registered new tserver with Master: a5ae2525dc7149a89c9d114e0f22d2bc (127.10.13.1:42463)
I20260925 02:31:32.438969 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:45097
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:32.460676 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44981
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:32.546017 12945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:32.546288 12945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:32.546329 12945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:32.547554 12945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:32.547633 12945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:32.547677 12945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:32.549206 12945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:32.549490 12945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:32.549746 12945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:32.554984 12951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:32.555078 12945 server_base.cc:1034] running on GCE node
W20260925 02:31:32.554956 12953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:32.554956 12950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:32.555517 12945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:32.555807 12945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:32.556933 12945 hybrid_clock.cc:648] HybridClock initialized: now 1790303492556915 us; error 27 us; skew 500 ppm
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:32.559744 12945 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:32.560226 12945 webserver.cc:460] Webserver started at http://127.10.13.2:40769/ using document root <none> and password file <none>
I20260925 02:31:32.560467 12945 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:32.560536 12945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:32.560725 12945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:32.561877 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "393051d0c7ac44c7969596ce884c9aad"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "d5ca109dcc5018b6f0c94048c20a0c5a"
server_key_iv: "af41ccc3a6bcb36dc591852ae8ba5e05"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.562261 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "393051d0c7ac44c7969596ce884c9aad"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "d5ca109dcc5018b6f0c94048c20a0c5a"
server_key_iv: "af41ccc3a6bcb36dc591852ae8ba5e05"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.563673 12945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:32.564476 12960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.564679 12945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:32.564785 12945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "393051d0c7ac44c7969596ce884c9aad"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "d5ca109dcc5018b6f0c94048c20a0c5a"
server_key_iv: "af41ccc3a6bcb36dc591852ae8ba5e05"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.564872 12945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:32.574474 12945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:32.576474 12945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:32.576649 12945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:32.576954 12945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:32.577311 12945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:32.577369 12945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.577412 12945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:32.577438 12945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.583441 12945 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:38145
I20260925 02:31:32.583534 13074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:38145 every 8 connection(s)
I20260925 02:31:32.583868 12945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:32.585856 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 12945
I20260925 02:31:32.585974 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:32.586218 10292 external_mini_cluster.cc:1428] Setting key ffe03ab7e67a329cdae36a62e8202670
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:32.594647 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:32.584679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:51761 (local address 127.10.13.62:44981)
0925 02:31:32.584746 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:32.584750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:32.585028 (+ 278us) server_negotiation.cc:407] Connection header received
0925 02:31:32.585616 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:32.585617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:32.585628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:32.585671 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:32.586880 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.587399 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.588796 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.589129 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.589200 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:32.591482 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:32.591484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:32.591484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:32.591494 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:32.593188 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:32.593592 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:32.593594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:32.593595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:32.593660 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:32.593903 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:32.593904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:32.593905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:32.594081 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:32.594160 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:32.594423 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 02:31:32.594515 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:31:32.594676 13077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:32.584669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44981 (local address 127.10.13.2:51761)
0925 02:31:32.584855 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:32.584942 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:32.585516 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:32.585808 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:32.585813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:32.586598 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:32.586808 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:32.586813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.587498 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.587499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:32.588702 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:32.588705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.589225 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.589226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:32.589334 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:32.589937 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:32.589946 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:32.591376 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:32.593276 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:32.593283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:32.593286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:32.593508 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:32.593732 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:32.593733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:32.593733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:32.593834 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:32.594152 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:32.594156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:32.594278 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 02:31:32.594362 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:31:32.594488 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260925 02:31:32.595340 13075 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44981
I20260925 02:31:32.595436 13075 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
I20260925 02:31:32.595644 13075 heartbeater.cc:511] Master 127.10.13.62:44981 requested a full tablet report, sending...
I20260925 02:31:32.596065 12747 ts_manager.cc:194] Registered new tserver with Master: 393051d0c7ac44c7969596ce884c9aad (127.10.13.2:38145)
I20260925 02:31:32.596418 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:51761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:32.616412 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44981
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:32.700762 13080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:32.701067 13080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:32.701097 13080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:32.702286 13080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:32.702323 13080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:32.702350 13080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:32.703784 13080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:32.703990 13080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:32.704187 13080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:32.709524 13088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:32.709502 13085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:32.709509 13086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:32.709573 13080 server_base.cc:1034] running on GCE node
I20260925 02:31:32.709935 13080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:32.710162 13080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:32.711305 13080 hybrid_clock.cc:648] HybridClock initialized: now 1790303492711294 us; error 33 us; skew 500 ppm
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:32.714025 13080 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:32.714460 13080 webserver.cc:460] Webserver started at http://127.10.13.3:46633/ using document root <none> and password file <none>
I20260925 02:31:32.714716 13080 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:32.714763 13080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:32.714877 13080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:32.715970 13080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "6ba4d06ce4c34245a6707a85cac8f4a0"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "130f2b0b3ad46701c89be727991fb9d9"
server_key_iv: "0f35150953ab96e32d2250074ad75236"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.716296 13080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "6ba4d06ce4c34245a6707a85cac8f4a0"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "130f2b0b3ad46701c89be727991fb9d9"
server_key_iv: "0f35150953ab96e32d2250074ad75236"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.717944 13080 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:31:32.718742 13095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.718935 13080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:32.719045 13080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "6ba4d06ce4c34245a6707a85cac8f4a0"
format_stamp: "Formatted at 2026-09-25 02:31:32 on dist-test-slave-q8mg"
server_key: "130f2b0b3ad46701c89be727991fb9d9"
server_key_iv: "0f35150953ab96e32d2250074ad75236"
server_key_version: "encryptionkey@0"
I20260925 02:31:32.719131 13080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:32.734208 13080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:32.736101 13080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:32.736282 13080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:32.736536 13080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:32.736858 13080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:32.736984 13080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.737025 13080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:32.737052 13080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:32.743058 13080 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:43561
I20260925 02:31:32.743108 13209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:43561 every 8 connection(s)
I20260925 02:31:32.743446 13080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:32.751839 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 13080
I20260925 02:31:32.751924 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:32.752167 10292 external_mini_cluster.cc:1428] Setting key 3925012110fe4d2be2b1cd0db33593f3
I20260925 02:31:32.753269 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:32.744269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:40295 (local address 127.10.13.62:44981)
0925 02:31:32.744326 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:32.744328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:32.744599 (+ 271us) server_negotiation.cc:407] Connection header received
0925 02:31:32.745221 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:32.745222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:32.745236 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:32.745277 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:32.746266 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.746789 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.747818 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.748016 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.748096 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:32.750314 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:32.750317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:32.750317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:32.750332 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:32.751520 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:32.751992 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:32.751999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:32.752000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:32.752117 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:32.752431 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:32.752432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:32.752432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:32.752608 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:32.752670 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:32.753074 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 02:31:32.753174 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:31:32.753505 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:32.744283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44981 (local address 127.10.13.3:40295)
0925 02:31:32.744461 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:32.744505 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:32.745101 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:32.745367 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:32.745371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:32.745960 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:32.746160 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:32.746166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.746891 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.746892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:32.747718 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:32.747720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:32.748104 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:32.748105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:32.748241 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:32.748757 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:32.748766 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:32.750200 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:32.751606 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:32.751614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:32.751630 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:32.751875 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:32.752204 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:32.752205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:32.752205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:32.752354 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:32.752714 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:32.752720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:32.752850 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 02:31:32.753038 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 02:31:32.753212 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260925 02:31:32.754017 13210 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44981
I20260925 02:31:32.754138 13210 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:32.754347 13210 heartbeater.cc:511] Master 127.10.13.62:44981 requested a full tablet report, sending...
I20260925 02:31:32.754755 12747 ts_manager.cc:194] Registered new tserver with Master: 6ba4d06ce4c34245a6707a85cac8f4a0 (127.10.13.3:43561)
I20260925 02:31:32.755178 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:40295
I20260925 02:31:32.764292 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:31:32.775874 12788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:32.774855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.10.13.62:44981)
0925 02:31:32.774931 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:32.774933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:32.775052 (+ 119us) server_negotiation.cc:407] Connection header received
0925 02:31:32.775108 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:32.775109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:32.775135 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:32.775189 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:32.775798 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56602: BlockingRecv error: recv got EOF from 127.0.0.1:56602 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:31:32.776705 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 12810
I20260925 02:31:32.781503 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 12945
I20260925 02:31:32.786501 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 13080
I20260925 02:31:32.791231 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 12715
2026-09-25T02:31:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1879 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[13222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[13222](info): set up 2 sockets
Sep 25 02:31:32 dist-test-slave-q8mg krb5kdc[13222](info): commencing operation
krb5kdc: starting...
W20260925 02:31:33.835337 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:33 dist-test-slave-q8mg krb5kdc[13222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:33.956794 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45517
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:45517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:34.039243 13235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:34.039453 13235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:34.039475 13235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:34.040670 13235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:34.040717 13235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:34.040731 13235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:34.040745 13235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:34.040756 13235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:34.042346 13235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:45517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:34.042621 13235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:34.042865 13235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:34.048245 13241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:34.048431 13235 server_base.cc:1034] running on GCE node
W20260925 02:31:34.048242 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:34.048269 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:34.048799 13235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:34.049160 13235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:34.050323 13235 hybrid_clock.cc:648] HybridClock initialized: now 1790303494050307 us; error 31 us; skew 500 ppm
Sep 25 02:31:34 dist-test-slave-q8mg krb5kdc[13222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303494, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:34.053100 13235 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:34.053568 13235 webserver.cc:460] Webserver started at http://127.10.13.62:46345/ using document root <none> and password file <none>
I20260925 02:31:34.053812 13235 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:34.053891 13235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:34.054067 13235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:34.055178 13235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "b848f015e09f405ba262ef49c5f32d7d"
format_stamp: "Formatted at 2026-09-25 02:31:34 on dist-test-slave-q8mg"
server_key: "eea07e7e49d40e91927ec81eac56f081"
server_key_iv: "5e57a9d2dddde2ed982ade3085b20fde"
server_key_version: "encryptionkey@0"
I20260925 02:31:34.055604 13235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "b848f015e09f405ba262ef49c5f32d7d"
format_stamp: "Formatted at 2026-09-25 02:31:34 on dist-test-slave-q8mg"
server_key: "eea07e7e49d40e91927ec81eac56f081"
server_key_iv: "5e57a9d2dddde2ed982ade3085b20fde"
server_key_version: "encryptionkey@0"
I20260925 02:31:34.057193 13235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:31:34.058033 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:34.058218 13235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:34.058333 13235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "b848f015e09f405ba262ef49c5f32d7d"
format_stamp: "Formatted at 2026-09-25 02:31:34 on dist-test-slave-q8mg"
server_key: "eea07e7e49d40e91927ec81eac56f081"
server_key_iv: "5e57a9d2dddde2ed982ade3085b20fde"
server_key_version: "encryptionkey@0"
I20260925 02:31:34.058413 13235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:34.068553 13235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:34.070596 13235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:34.070766 13235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:34.074928 13235 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:45517
I20260925 02:31:34.074961 13302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:45517 every 8 connection(s)
I20260925 02:31:34.075479 13235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:34.076192 13303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:34.078625 13303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d: Bootstrap starting.
I20260925 02:31:34.079263 13303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:34.079578 13303 log.cc:824] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:34.080389 13303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d: No bootstrap required, opened a new log
I20260925 02:31:34.081645 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 13235
I20260925 02:31:34.081763 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:34.081930 10292 external_mini_cluster.cc:1428] Setting key c48a545463fe24bbb854e234867cdaab
I20260925 02:31:34.082233 13303 raft_consensus.cc:348] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848f015e09f405ba262ef49c5f32d7d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45517 } }
I20260925 02:31:34.082427 13303 raft_consensus.cc:374] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:34.082472 13303 raft_consensus.cc:729] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b848f015e09f405ba262ef49c5f32d7d, State: Initialized, Role: FOLLOWER
I20260925 02:31:34.082615 13303 consensus_queue.cc:260] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848f015e09f405ba262ef49c5f32d7d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45517 } }
I20260925 02:31:34.082716 13303 raft_consensus.cc:388] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:34.082752 13303 raft_consensus.cc:482] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:34.082808 13303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:34.083357 13303 raft_consensus.cc:504] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848f015e09f405ba262ef49c5f32d7d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45517 } }
I20260925 02:31:34.083485 13303 leader_election.cc:302] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b848f015e09f405ba262ef49c5f32d7d; no voters:
I20260925 02:31:34.083678 13303 leader_election.cc:288] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:34.083706 13308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:34.083808 13308 raft_consensus.cc:686] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [term 1 LEADER]: Becoming Leader. State: Replica: b848f015e09f405ba262ef49c5f32d7d, State: Running, Role: LEADER
I20260925 02:31:34.083891 13308 consensus_queue.cc:237] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848f015e09f405ba262ef49c5f32d7d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45517 } }
I20260925 02:31:34.083953 13303 sys_catalog.cc:564] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:34.084236 13308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b848f015e09f405ba262ef49c5f32d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848f015e09f405ba262ef49c5f32d7d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45517 } } }
I20260925 02:31:34.084309 13308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:34.084429 13308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b848f015e09f405ba262ef49c5f32d7d. Latest consensus state: current_term: 1 leader_uuid: "b848f015e09f405ba262ef49c5f32d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848f015e09f405ba262ef49c5f32d7d" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45517 } } }
I20260925 02:31:34.084497 13308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:34.084650 13313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:34.085382 13313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:34.087230 13313 catalog_manager.cc:1347] Generated new cluster ID: 363a6b7d398e4892a14ea2417089ba35
I20260925 02:31:34.087291 13313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:31:34 dist-test-slave-q8mg krb5kdc[13222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:34.094524 13313 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:34.095064 13313 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:34.099217 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:34.082492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.10.13.62:45517)
0925 02:31:34.083183 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:34.083191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:34.083217 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:31:34.083316 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:34.083321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:34.083364 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:34.083539 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:34.084394 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:34.085891 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:34.086913 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:34.087460 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:34.087588 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:34.090062 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:34.090067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:34.090067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:34.090081 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:34.097439 (+ 7358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:34.097936 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:34.097940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:34.097941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:34.098063 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:34.098278 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:34.098278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:34.098279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:34.098555 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:34.098690 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:34.098820 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 02:31:34.098965 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":631,"thread_start_us":598,"threads_started":1}
I20260925 02:31:34.100342 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:31:34.101676 13313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b848f015e09f405ba262ef49c5f32d7d: Generated new TSK 0
I20260925 02:31:34.101847 13313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:31:34.115237 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:34.111196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.10.13.62:45517)
0925 02:31:34.111272 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:34.111274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:34.111388 (+ 114us) server_negotiation.cc:407] Connection header received
0925 02:31:34.111495 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:34.111495 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:34.111507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:34.111559 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:34.112037 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:34.112647 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:34.113769 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:34.113981 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:34.114033 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:34.114736 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:34.114737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:34.114738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:34.114951 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:34.114968 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:34.114974 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:34.115021 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:34.115107 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 02:31:34.115133 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:31:34.117681 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 13235
2026-09-25T02:31:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1324 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:34 dist-test-slave-q8mg krb5kdc[13336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:34 dist-test-slave-q8mg krb5kdc[13336](info): set up 2 sockets
Sep 25 02:31:34 dist-test-slave-q8mg krb5kdc[13336](info): commencing operation
krb5kdc: starting...
W20260925 02:31:35.167309 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:35.291378 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:42855
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:42855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:35.374130 13350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:35.374366 13350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:35.374406 13350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:35.375581 13350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:35.375653 13350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:35.375677 13350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:35.375712 13350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:35.375741 13350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:35.377290 13350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:42855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:42855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:35.377570 13350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:35.377812 13350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:35.382929 13355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:35.382915 13356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:35.382958 13358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:35.383296 13350 server_base.cc:1034] running on GCE node
I20260925 02:31:35.383512 13350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:35.383780 13350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:35.384959 13350 hybrid_clock.cc:648] HybridClock initialized: now 1790303495384941 us; error 42 us; skew 500 ppm
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:35.387586 13350 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:35.388120 13350 webserver.cc:460] Webserver started at http://127.10.13.62:43691/ using document root <none> and password file <none>
I20260925 02:31:35.388388 13350 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:35.388434 13350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:35.388563 13350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:35.389719 13350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "15d0037b79e349b6b2decfb6f832b7ef"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "f73a848ac6d84168cd548da65696191c"
server_key_iv: "508fb04bfd892594804e871a8605b4da"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.390089 13350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "15d0037b79e349b6b2decfb6f832b7ef"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "f73a848ac6d84168cd548da65696191c"
server_key_iv: "508fb04bfd892594804e871a8605b4da"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.391507 13350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:31:35.392400 13365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.392570 13350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:35.392681 13350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "15d0037b79e349b6b2decfb6f832b7ef"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "f73a848ac6d84168cd548da65696191c"
server_key_iv: "508fb04bfd892594804e871a8605b4da"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.392756 13350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:35.399730 13350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:35.401688 13350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:35.401850 13350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:35.405977 13350 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:42855
I20260925 02:31:35.406008 13417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:42855 every 8 connection(s)
I20260925 02:31:35.406406 13350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:35.406579 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 13350
I20260925 02:31:35.406703 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:35.406930 10292 external_mini_cluster.cc:1428] Setting key dd10aea0ecf26b42e77ea78c7cbc3336
I20260925 02:31:35.407752 13418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:35.410671 13418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef: Bootstrap starting.
I20260925 02:31:35.411381 13418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:35.411682 13418 log.cc:824] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:35.412515 13418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef: No bootstrap required, opened a new log
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:35.414821 13418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d0037b79e349b6b2decfb6f832b7ef" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42855 } }
I20260925 02:31:35.415009 13418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:35.415061 13418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d0037b79e349b6b2decfb6f832b7ef, State: Initialized, Role: FOLLOWER
I20260925 02:31:35.415261 13418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d0037b79e349b6b2decfb6f832b7ef" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42855 } }
I20260925 02:31:35.415365 13418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:35.415403 13418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:35.415454 13418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:35.415989 13418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d0037b79e349b6b2decfb6f832b7ef" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42855 } }
I20260925 02:31:35.416119 13418 leader_election.cc:302] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15d0037b79e349b6b2decfb6f832b7ef; no voters:
I20260925 02:31:35.416304 13418 leader_election.cc:288] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:35.416360 13423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:35.416520 13423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [term 1 LEADER]: Becoming Leader. State: Replica: 15d0037b79e349b6b2decfb6f832b7ef, State: Running, Role: LEADER
I20260925 02:31:35.416595 13418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:35.416714 13423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d0037b79e349b6b2decfb6f832b7ef" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42855 } }
I20260925 02:31:35.418678 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:35.407498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.10.13.62:42855)
0925 02:31:35.407678 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:35.407688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:35.407721 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:31:35.408221 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:35.408227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:35.408276 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:35.408391 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:35.409223 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.410162 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.411235 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.411773 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.411865 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:35.414566 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:35.414570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:35.414571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:35.414583 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:35.416701 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.417122 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.417125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.417126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.417244 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.417514 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.417515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.417515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.417885 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:35.418121 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:35.418208 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 02:31:35.418458 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260925 02:31:35.418678 13424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d0037b79e349b6b2decfb6f832b7ef. Latest consensus state: current_term: 1 leader_uuid: "15d0037b79e349b6b2decfb6f832b7ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d0037b79e349b6b2decfb6f832b7ef" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42855 } } }
I20260925 02:31:35.418679 13425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15d0037b79e349b6b2decfb6f832b7ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d0037b79e349b6b2decfb6f832b7ef" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42855 } } }
I20260925 02:31:35.419023 13424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:35.419031 13425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef [sys.catalog]: This master's current role is: LEADER
W20260925 02:31:35.419516 13437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:31:35.419575 13437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:31:35.419692 13440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:35.419867 13440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:35.421386 13440 catalog_manager.cc:1347] Generated new cluster ID: 3dbf3d4f7ad94b6cb7391115c9d55d9e
I20260925 02:31:35.421437 13440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:35.443346 13440 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:35.443921 13440 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:35.453048 13440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15d0037b79e349b6b2decfb6f832b7ef: Generated new TSK 0
I20260925 02:31:35.453281 13440 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:35.491333 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42855
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:35.596612 13444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:35.596921 13444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:35.596963 13444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:35.598170 13444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:35.598212 13444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:35.598238 13444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:35.599611 13444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:35.599812 13444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:35.600001 13444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:35.604951 13452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:35.605049 13444 server_base.cc:1034] running on GCE node
W20260925 02:31:35.604951 13449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:35.605120 13450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:35.605379 13444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:35.605638 13444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:35.606798 13444 hybrid_clock.cc:648] HybridClock initialized: now 1790303495606797 us; error 36 us; skew 500 ppm
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:35.609277 13444 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:35.609738 13444 webserver.cc:460] Webserver started at http://127.10.13.1:37343/ using document root <none> and password file <none>
I20260925 02:31:35.610013 13444 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:35.610080 13444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:35.610229 13444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:35.611277 13444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "9003ed17be2a4c80b902eae2a303cffe"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "b173b66646b02bd81e4ce485a1c3f4c9"
server_key_iv: "505a7ff8617d8752b87a5a34d9f6c0e6"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.611634 13444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "9003ed17be2a4c80b902eae2a303cffe"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "b173b66646b02bd81e4ce485a1c3f4c9"
server_key_iv: "505a7ff8617d8752b87a5a34d9f6c0e6"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.613024 13444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:31:35.613829 13459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.614039 13444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:35.614145 13444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "9003ed17be2a4c80b902eae2a303cffe"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "b173b66646b02bd81e4ce485a1c3f4c9"
server_key_iv: "505a7ff8617d8752b87a5a34d9f6c0e6"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.614234 13444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:35.626173 13444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:35.627996 13444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:35.628168 13444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:35.628401 13444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:35.628701 13444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:35.628755 13444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.628798 13444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:35.628826 13444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.634934 13444 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:42321
I20260925 02:31:35.634979 13573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:42321 every 8 connection(s)
I20260925 02:31:35.635306 13444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:35.637804 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 13444
I20260925 02:31:35.637892 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:35.638119 10292 external_mini_cluster.cc:1428] Setting key 9b599c4c6c9a01f23466ceaf8be9dee3
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:35.645608 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:35.636040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:46199 (local address 127.10.13.62:42855)
0925 02:31:35.636157 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:35.636159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:35.636369 (+ 210us) server_negotiation.cc:407] Connection header received
0925 02:31:35.636980 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:35.636982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:35.636996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:35.637044 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:35.638059 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.638641 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.639706 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.640045 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.640140 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:35.642538 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:35.642540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:35.642540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:35.642549 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:35.644182 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.644581 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.644583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.644583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.644647 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.644908 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.644909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.644909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.645051 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:35.645125 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:35.645382 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 02:31:35.645479 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:31:35.645642 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:35.636042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:42855 (local address 127.10.13.1:46199)
0925 02:31:35.636226 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:35.636276 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:35.636833 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:35.637112 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:35.637115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:35.637677 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:35.637898 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:35.637904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.638811 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.638812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:35.639617 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:35.639619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.640150 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.640151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:35.640289 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:35.640814 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:35.640823 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:35.642435 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:35.644274 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:35.644281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:35.644283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:35.644502 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:35.644717 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:35.644718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:35.644718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:35.644817 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:35.645120 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:35.645126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:35.645248 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 02:31:35.645334 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:31:35.645456 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260925 02:31:35.646224 13574 heartbeater.cc:347] Connected to a master server at 127.10.13.62:42855
I20260925 02:31:35.646337 13574 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:35.646566 13574 heartbeater.cc:511] Master 127.10.13.62:42855 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
I20260925 02:31:35.647153 13382 ts_manager.cc:194] Registered new tserver with Master: 9003ed17be2a4c80b902eae2a303cffe (127.10.13.1:42321)
I20260925 02:31:35.647794 13382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:46199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:35.664716 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42855
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:35.751296 13579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:35.751504 13579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:35.751529 13579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:35.752640 13579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:35.752676 13579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:35.752701 13579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:35.754194 13579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:35.754391 13579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:35.754582 13579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:35.759553 13584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:35.759660 13585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:35.759719 13587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:35.759881 13579 server_base.cc:1034] running on GCE node
I20260925 02:31:35.760051 13579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:35.760308 13579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:35.761461 13579 hybrid_clock.cc:648] HybridClock initialized: now 1790303495761437 us; error 38 us; skew 500 ppm
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:35.763873 13579 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:35.764374 13579 webserver.cc:460] Webserver started at http://127.10.13.2:33073/ using document root <none> and password file <none>
I20260925 02:31:35.764604 13579 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:35.764652 13579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:35.764822 13579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:35.765937 13579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "3ca38870e36a4612b8fa0c7861961912"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "a7793c4df20a7323a7f24cf37a454cd1"
server_key_iv: "5f862f1199486cba949d8852c00329dd"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.766304 13579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "3ca38870e36a4612b8fa0c7861961912"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "a7793c4df20a7323a7f24cf37a454cd1"
server_key_iv: "5f862f1199486cba949d8852c00329dd"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.767637 13579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:35.768380 13594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.768553 13579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:35.768657 13579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "3ca38870e36a4612b8fa0c7861961912"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "a7793c4df20a7323a7f24cf37a454cd1"
server_key_iv: "5f862f1199486cba949d8852c00329dd"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.768741 13579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:35.792641 13579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:35.794740 13579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:35.794862 13579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:35.795064 13579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:35.795470 13579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:35.795509 13579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.795552 13579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:35.795578 13579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.801748 13579 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:33323
I20260925 02:31:35.801798 13708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:33323 every 8 connection(s)
I20260925 02:31:35.802130 13579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:35.810513 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 13579
I20260925 02:31:35.810652 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:35.810902 10292 external_mini_cluster.cc:1428] Setting key 8d531667d82059098dd866d9506f66fb
I20260925 02:31:35.811339 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:35.802829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:41429 (local address 127.10.13.62:42855)
0925 02:31:35.802880 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:35.802883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:35.803070 (+ 187us) server_negotiation.cc:407] Connection header received
0925 02:31:35.803570 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:35.803571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:35.803581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:35.803648 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:35.804565 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.805142 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.806132 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.806308 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.806356 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:35.808488 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:35.808490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:35.808490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:35.808497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:35.809807 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.810206 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.810207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.810207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.810263 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.810522 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.810524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.810524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.810699 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:35.810792 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:35.811149 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 02:31:35.811233 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:31:35.811589 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:35.802754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:42855 (local address 127.10.13.2:41429)
0925 02:31:35.802924 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:35.802967 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:35.803480 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:35.803721 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:35.803724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:35.804288 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:35.804483 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:35.804487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.805222 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.805223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:35.806053 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:35.806056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.806379 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.806379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:35.806479 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:35.807021 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:35.807029 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:35.808401 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:35.809880 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:35.809887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:35.809889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:35.810130 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:35.810336 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:35.810336 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:35.810337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:35.810439 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:35.810798 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:35.810806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:35.810986 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 02:31:35.811111 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 02:31:35.811314 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260925 02:31:35.811991 13709 heartbeater.cc:347] Connected to a master server at 127.10.13.62:42855
I20260925 02:31:35.812093 13709 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:35.812294 13709 heartbeater.cc:511] Master 127.10.13.62:42855 requested a full tablet report, sending...
I20260925 02:31:35.812682 13382 ts_manager.cc:194] Registered new tserver with Master: 3ca38870e36a4612b8fa0c7861961912 (127.10.13.2:33323)
I20260925 02:31:35.813154 13382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:41429
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:35.837656 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42855
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:35.921116 13714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:35.921319 13714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:35.921341 13714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:35.922442 13714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:35.922479 13714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:35.922504 13714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:35.923872 13714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:35.924079 13714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:35.924266 13714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:35.929268 13720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:35.929265 13719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:35.929265 13722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:35.929582 13714 server_base.cc:1034] running on GCE node
I20260925 02:31:35.929739 13714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:35.929965 13714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:35.931102 13714 hybrid_clock.cc:648] HybridClock initialized: now 1790303495931086 us; error 31 us; skew 500 ppm
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:35.933571 13714 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:35.934036 13714 webserver.cc:460] Webserver started at http://127.10.13.3:43509/ using document root <none> and password file <none>
I20260925 02:31:35.934253 13714 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:35.934294 13714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:35.934381 13714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:35.935369 13714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "f901a431ff4845a78ea0c0da132c1f51"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "960f9e57647ad15a44f6d2017740f766"
server_key_iv: "5940174deb556d3603ee2576a79645cf"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.935659 13714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "f901a431ff4845a78ea0c0da132c1f51"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "960f9e57647ad15a44f6d2017740f766"
server_key_iv: "5940174deb556d3603ee2576a79645cf"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.936995 13714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:35.937747 13729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.937919 13714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:35.938030 13714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "f901a431ff4845a78ea0c0da132c1f51"
format_stamp: "Formatted at 2026-09-25 02:31:35 on dist-test-slave-q8mg"
server_key: "960f9e57647ad15a44f6d2017740f766"
server_key_iv: "5940174deb556d3603ee2576a79645cf"
server_key_version: "encryptionkey@0"
I20260925 02:31:35.938133 13714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:35.948311 13714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:35.950774 13714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:35.950951 13714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:35.951218 13714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:35.951582 13714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:35.951642 13714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.951683 13714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:35.951713 13714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:35.958098 13714 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:43565
I20260925 02:31:35.958137 13843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:43565 every 8 connection(s)
I20260925 02:31:35.958470 13714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:35.962515 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 13714
I20260925 02:31:35.962623 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:35.962824 10292 external_mini_cluster.cc:1428] Setting key bc25b47d4e50fb706edcf82b5d6add4c
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:35.967933 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:35.959170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:53951 (local address 127.10.13.62:42855)
0925 02:31:35.959252 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:35.959255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:35.959429 (+ 174us) server_negotiation.cc:407] Connection header received
0925 02:31:35.959912 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:35.959913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:35.959924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:35.959977 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:35.960946 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.961461 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.962458 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.962762 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.962842 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:35.965275 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:35.965276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:35.965277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:35.965284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:35.966762 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.967130 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.967132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.967132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.967190 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.967405 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.967406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.967406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.967526 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:35.967576 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:35.967807 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 02:31:35.967868 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:31:35.968195 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:35.959128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:42855 (local address 127.10.13.3:53951)
0925 02:31:35.959309 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:35.959346 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:35.959829 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:35.960036 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:35.960040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:35.960603 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:35.960802 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:35.960807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.961544 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.961545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:35.962378 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:35.962380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.962858 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.962877 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:35.963067 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:35.963665 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:35.963682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:35.965167 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:35.966834 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:35.966840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:35.966842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:35.967062 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:35.967255 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:35.967256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:35.967256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:35.967352 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:35.967583 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:35.967587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:35.967701 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 02:31:35.967783 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:31:35.967901 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260925 02:31:35.968729 13844 heartbeater.cc:347] Connected to a master server at 127.10.13.62:42855
I20260925 02:31:35.968847 13844 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:35.969058 13844 heartbeater.cc:511] Master 127.10.13.62:42855 requested a full tablet report, sending...
I20260925 02:31:35.969460 13382 ts_manager.cc:194] Registered new tserver with Master: f901a431ff4845a78ea0c0da132c1f51 (127.10.13.3:43565)
I20260925 02:31:35.969806 13382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:53951
I20260925 02:31:35.974714 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 02:31:35 dist-test-slave-q8mg krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303495, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:35.997609 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:35.990003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.10.13.62:42855)
0925 02:31:35.990095 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:35.990097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:35.990213 (+ 116us) server_negotiation.cc:407] Connection header received
0925 02:31:35.990312 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:35.990313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:35.990324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:35.990397 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:35.990983 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.991498 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.992296 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:35.992504 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:35.992597 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:35.995023 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:35.995025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:35.995025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:35.995033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:35.996591 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.996940 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.996942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.996942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.996997 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:35.997165 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:35.997165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:35.997165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:35.997316 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:35.997390 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:35.997480 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 02:31:35.997538 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 02:31:35.997886 13382 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38600
I20260925 02:31:35.998358 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 13444
I20260925 02:31:36.004772 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 13579
I20260925 02:31:36.009560 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 13714
I20260925 02:31:36.013963 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 13350
2026-09-25T02:31:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1898 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:36 dist-test-slave-q8mg krb5kdc[13856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:36 dist-test-slave-q8mg krb5kdc[13856](info): set up 2 sockets
Sep 25 02:31:36 dist-test-slave-q8mg krb5kdc[13856](info): commencing operation
krb5kdc: starting...
W20260925 02:31:37.063396 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:37.185809 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40523
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:40523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:37.267831 13869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:37.268046 13869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:37.268069 13869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:37.269219 13869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:37.269255 13869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:37.269270 13869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:37.269282 13869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:37.269294 13869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:37.270779 13869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:40523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:37.270977 13869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:37.271148 13869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:37.276019 13876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:37.275997 13878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:37.276314 13869 server_base.cc:1034] running on GCE node
W20260925 02:31:37.275998 13874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:37.276644 13869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:37.276978 13869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:37.278195 13869 hybrid_clock.cc:648] HybridClock initialized: now 1790303497278192 us; error 40 us; skew 500 ppm
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:37.280963 13869 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:37.281502 13869 webserver.cc:460] Webserver started at http://127.10.13.62:45473/ using document root <none> and password file <none>
I20260925 02:31:37.281759 13869 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:37.281831 13869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:37.281981 13869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:37.283082 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "4b7e22fb64cd45a78b1507721200460c"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "ad811db379ff1cb7fd01729e91b353c3"
server_key_iv: "2dc9c745597b5332d2820bfa55153c1d"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.283449 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "4b7e22fb64cd45a78b1507721200460c"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "ad811db379ff1cb7fd01729e91b353c3"
server_key_iv: "2dc9c745597b5332d2820bfa55153c1d"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.284829 13869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:37.285679 13885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.285856 13869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:37.285954 13869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "4b7e22fb64cd45a78b1507721200460c"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "ad811db379ff1cb7fd01729e91b353c3"
server_key_iv: "2dc9c745597b5332d2820bfa55153c1d"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.286087 13869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:37.333633 13869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:37.336081 13869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:37.336256 13869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:37.340723 13869 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:40523
I20260925 02:31:37.340771 13937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:40523 every 8 connection(s)
I20260925 02:31:37.341316 13869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:37.342109 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:37.344659 13938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c: Bootstrap starting.
I20260925 02:31:37.345364 13938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:37.345705 13938 log.cc:824] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:37.346505 13938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c: No bootstrap required, opened a new log
I20260925 02:31:37.348349 13938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e22fb64cd45a78b1507721200460c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40523 } }
I20260925 02:31:37.348506 13938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:37.348532 13938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7e22fb64cd45a78b1507721200460c, State: Initialized, Role: FOLLOWER
I20260925 02:31:37.348651 13938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e22fb64cd45a78b1507721200460c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40523 } }
I20260925 02:31:37.348747 13938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:37.348779 13938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:37.348830 13938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:37.349839 13938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e22fb64cd45a78b1507721200460c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40523 } }
I20260925 02:31:37.349977 13938 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7e22fb64cd45a78b1507721200460c; no voters:
I20260925 02:31:37.350180 13938 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:37.350237 13941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:37.350414 13941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [term 1 LEADER]: Becoming Leader. State: Replica: 4b7e22fb64cd45a78b1507721200460c, State: Running, Role: LEADER
I20260925 02:31:37.350485 13938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:37.350558 13941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e22fb64cd45a78b1507721200460c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40523 } }
I20260925 02:31:37.350965 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7e22fb64cd45a78b1507721200460c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e22fb64cd45a78b1507721200460c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40523 } } }
I20260925 02:31:37.351092 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7e22fb64cd45a78b1507721200460c. Latest consensus state: current_term: 1 leader_uuid: "4b7e22fb64cd45a78b1507721200460c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e22fb64cd45a78b1507721200460c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40523 } } }
I20260925 02:31:37.351156 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:37.351236 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:37.351495 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 13869
I20260925 02:31:37.351622 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:37.351846 10292 external_mini_cluster.cc:1428] Setting key 87ab379953d5369dd72b58b4bb9979e9
W20260925 02:31:37.353451 13959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:31:37.353518 13959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:31:37.353574 13950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:37.353787 13950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:37.355574 13950 catalog_manager.cc:1347] Generated new cluster ID: 38535c847b5d4a919218113c9b27cd14
I20260925 02:31:37.355634 13950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:37.363399 13950 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:37.364079 13950 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:37.364432 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.352426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.10.13.62:40523)
0925 02:31:37.352629 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:37.352638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:37.352669 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:31:37.352778 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:37.352783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:37.352834 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:37.353003 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:37.353807 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.355378 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.356320 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.356858 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.357040 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.359585 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:37.359590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:37.359590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:37.359604 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:37.362124 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.362548 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.362552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.362552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.362655 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.362904 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.362906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.362906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.363284 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:37.363915 (+ 631us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:37.363999 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:31:37.364183 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260925 02:31:37.367057 13950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7e22fb64cd45a78b1507721200460c: Generated new TSK 0
I20260925 02:31:37.367268 13950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:37.397840 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40523
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:37.482050 13964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:37.482327 13964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:37.482393 13964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:37.483574 13964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:37.483615 13964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:37.483642 13964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:37.485234 13964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:37.485486 13964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:37.485723 13964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:37.491369 13969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:37.491386 13964 server_base.cc:1034] running on GCE node
W20260925 02:31:37.491369 13972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:37.491822 13970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:37.492058 13964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:37.492336 13964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:37.493497 13964 hybrid_clock.cc:648] HybridClock initialized: now 1790303497493476 us; error 37 us; skew 500 ppm
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:37.495999 13964 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:37.496628 13964 webserver.cc:460] Webserver started at http://127.10.13.1:43601/ using document root <none> and password file <none>
I20260925 02:31:37.496908 13964 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:37.496974 13964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:37.497129 13964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:37.498211 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "bc45a228ac784a8da3e3f3ef12ec9913"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "c0c31f8d795001663a14e500afc0771b"
server_key_iv: "2c5d8f47fa1545a956998a4ea7a74720"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.498576 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "bc45a228ac784a8da3e3f3ef12ec9913"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "c0c31f8d795001663a14e500afc0771b"
server_key_iv: "2c5d8f47fa1545a956998a4ea7a74720"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.499958 13964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:37.500665 13979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.500895 13964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:37.500983 13964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "bc45a228ac784a8da3e3f3ef12ec9913"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "c0c31f8d795001663a14e500afc0771b"
server_key_iv: "2c5d8f47fa1545a956998a4ea7a74720"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.501039 13964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:37.517252 13964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:37.519277 13964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:37.519407 13964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:37.519599 13964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:37.519858 13964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:37.519888 13964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.519909 13964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:37.519922 13964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.525746 13964 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:38187
I20260925 02:31:37.525832 14092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:38187 every 8 connection(s)
I20260925 02:31:37.526161 13964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:37.533290 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 13964
I20260925 02:31:37.533432 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:37.533651 10292 external_mini_cluster.cc:1428] Setting key eae935a7537a2b4c103ecf2a85ea5d31
I20260925 02:31:37.536751 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.526922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:36367 (local address 127.10.13.62:40523)
0925 02:31:37.526990 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:37.526992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:37.527212 (+ 220us) server_negotiation.cc:407] Connection header received
0925 02:31:37.528107 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:37.528108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:37.528117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:37.528176 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:37.528746 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.529351 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.530313 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.530531 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.530623 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.533381 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:37.533383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:37.533384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:37.533392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:37.535166 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.535557 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.535559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.535559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.535628 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.535995 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.535997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.535997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.536143 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:37.536235 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:37.536561 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 02:31:37.536656 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:31:37.537043 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.526881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40523 (local address 127.10.13.1:36367)
0925 02:31:37.527070 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:37.527114 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:37.528020 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:37.528232 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:37.528235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:37.528475 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:37.528678 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:37.528682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.529430 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.529431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:37.530231 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:37.530233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.530686 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.530688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:37.530902 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.531498 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:37.531509 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:37.533272 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:37.535250 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:37.535257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:37.535260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:37.535482 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:37.535720 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:37.535721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:37.535721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:37.535893 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:37.536225 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:37.536234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:37.536394 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 02:31:37.536522 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 02:31:37.536697 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260925 02:31:37.537570 14093 heartbeater.cc:347] Connected to a master server at 127.10.13.62:40523
I20260925 02:31:37.537683 14093 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:37.537884 14093 heartbeater.cc:511] Master 127.10.13.62:40523 requested a full tablet report, sending...
I20260925 02:31:37.538552 13902 ts_manager.cc:194] Registered new tserver with Master: bc45a228ac784a8da3e3f3ef12ec9913 (127.10.13.1:38187)
I20260925 02:31:37.539228 13902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:36367
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:37.560233 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40523
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:37.641387 14098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:37.641626 14098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:37.641678 14098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:37.642841 14098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:37.642910 14098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:37.642954 14098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:37.644507 14098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:37.644794 14098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:37.645063 14098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:37.650388 14104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:37.650398 14106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:37.650396 14103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:37.650429 14098 server_base.cc:1034] running on GCE node
I20260925 02:31:37.650784 14098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:37.651047 14098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:37.652220 14098 hybrid_clock.cc:648] HybridClock initialized: now 1790303497652204 us; error 30 us; skew 500 ppm
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:37.654654 14098 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:37.655110 14098 webserver.cc:460] Webserver started at http://127.10.13.2:33629/ using document root <none> and password file <none>
I20260925 02:31:37.655349 14098 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:37.655417 14098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:37.655571 14098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:37.656598 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "b6e5b6f6ba10452698936db07cf17c31"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "bfc36b483d1a7eab22252f80e89f9880"
server_key_iv: "9252257dfca840ba961b11b3e3175a94"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.657029 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "b6e5b6f6ba10452698936db07cf17c31"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "bfc36b483d1a7eab22252f80e89f9880"
server_key_iv: "9252257dfca840ba961b11b3e3175a94"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.658346 14098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:37.659083 14113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.659260 14098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:37.659360 14098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "b6e5b6f6ba10452698936db07cf17c31"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "bfc36b483d1a7eab22252f80e89f9880"
server_key_iv: "9252257dfca840ba961b11b3e3175a94"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.659442 14098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:37.673784 14098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:37.675910 14098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:37.676079 14098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:37.676313 14098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:37.676632 14098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:37.676694 14098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.676738 14098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:37.676765 14098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.682827 14098 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:39721
I20260925 02:31:37.682898 14228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:39721 every 8 connection(s)
I20260925 02:31:37.683234 14098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:37.685063 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 14098
I20260925 02:31:37.685177 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:37.685371 10292 external_mini_cluster.cc:1428] Setting key 95e9416217305481080f05aac2b5b2aa
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:37.694020 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.683932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40523 (local address 127.10.13.2:53995)
0925 02:31:37.684151 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:37.684203 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:37.685246 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:37.685502 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:37.685506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:37.685923 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:37.686198 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:37.686205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.686901 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.686902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:37.687847 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:37.687849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.688269 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.688270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:37.688415 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.688955 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:37.688967 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:37.690725 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:37.692548 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:37.692555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:37.692556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:37.692927 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:37.693133 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:37.693133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:37.693134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:37.693242 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:37.693535 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:37.693539 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:37.693656 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 02:31:37.693729 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:31:37.693845 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260925 02:31:37.694037 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.684015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:53995 (local address 127.10.13.62:40523)
0925 02:31:37.684136 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:37.684139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:37.684305 (+ 166us) server_negotiation.cc:407] Connection header received
0925 02:31:37.685349 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:37.685350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:37.685363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:37.685430 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:37.686311 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.686806 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.687955 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.688200 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.688261 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.690820 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:37.690823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:37.690823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:37.690831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:37.692465 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.693002 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.693004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.693005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.693068 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.693322 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.693323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.693323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.693470 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:37.693542 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:37.693800 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 02:31:37.693894 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
I20260925 02:31:37.694631 14229 heartbeater.cc:347] Connected to a master server at 127.10.13.62:40523
I20260925 02:31:37.694742 14229 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:37.694953 14229 heartbeater.cc:511] Master 127.10.13.62:40523 requested a full tablet report, sending...
I20260925 02:31:37.695307 13902 ts_manager.cc:194] Registered new tserver with Master: b6e5b6f6ba10452698936db07cf17c31 (127.10.13.2:39721)
I20260925 02:31:37.695806 13902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:53995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:37.712808 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40523
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:37.794961 14234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:37.795204 14234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:37.795243 14234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:37.796396 14234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:37.796461 14234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:37.796506 14234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:37.798261 14234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:40523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:37.798543 14234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:37.798789 14234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:37.804569 14240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:37.804733 14239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:37.804636 14234 server_base.cc:1034] running on GCE node
W20260925 02:31:37.804571 14242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:37.805135 14234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:37.805439 14234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:37.806617 14234 hybrid_clock.cc:648] HybridClock initialized: now 1790303497806605 us; error 26 us; skew 500 ppm
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:37.809129 14234 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:37.809640 14234 webserver.cc:460] Webserver started at http://127.10.13.3:41987/ using document root <none> and password file <none>
I20260925 02:31:37.809897 14234 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:37.809968 14234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:37.810125 14234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:37.811215 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "ea5f90f4791445ea92955a3b7a198e7f"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "2d20ebfe70481889317e72aa8254894e"
server_key_iv: "ea6584a82008e8ca1d010ecee685aabd"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.811575 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "ea5f90f4791445ea92955a3b7a198e7f"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "2d20ebfe70481889317e72aa8254894e"
server_key_iv: "ea6584a82008e8ca1d010ecee685aabd"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.813005 14234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:37.813766 14249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.813947 14234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:37.814051 14234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "ea5f90f4791445ea92955a3b7a198e7f"
format_stamp: "Formatted at 2026-09-25 02:31:37 on dist-test-slave-q8mg"
server_key: "2d20ebfe70481889317e72aa8254894e"
server_key_iv: "ea6584a82008e8ca1d010ecee685aabd"
server_key_version: "encryptionkey@0"
I20260925 02:31:37.814138 14234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:37.833019 14234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:37.834991 14234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:37.835170 14234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:37.835423 14234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:37.835741 14234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:37.835796 14234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.835841 14234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:37.835865 14234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:37.842089 14234 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:38529
I20260925 02:31:37.842136 14363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:38529 every 8 connection(s)
I20260925 02:31:37.842502 14234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:37.847671 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 14234
I20260925 02:31:37.847767 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:37.847980 10292 external_mini_cluster.cc:1428] Setting key 070ac1d45a6232a31b545880a87ea364
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:37.852061 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.843311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:59439 (local address 127.10.13.62:40523)
0925 02:31:37.843355 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:37.843357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:37.843546 (+ 189us) server_negotiation.cc:407] Connection header received
0925 02:31:37.844474 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:37.844475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:37.844485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:37.844548 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:37.845239 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.845738 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.846744 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.846935 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.846991 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.849522 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:37.849523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:37.849524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:37.849530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:37.850794 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.851216 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.851217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.851218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.851275 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.851490 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.851490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.851491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.851616 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:37.851694 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:37.851933 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 02:31:37.851999 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:31:37.852312 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.843255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40523 (local address 127.10.13.3:59439)
0925 02:31:37.843424 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:37.843468 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:37.844377 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:37.844624 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:37.844628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:37.844955 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:37.845161 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:37.845167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.845822 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.845823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:37.846670 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:37.846673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.847012 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.847012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:37.847117 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.847674 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:37.847683 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:37.849437 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:37.850864 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:37.850871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:37.850872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:37.851123 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:37.851341 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:37.851341 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:37.851342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:37.851446 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:37.851682 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:37.851689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:37.851815 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 02:31:37.851899 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:31:37.852017 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260925 02:31:37.852918 14364 heartbeater.cc:347] Connected to a master server at 127.10.13.62:40523
I20260925 02:31:37.853012 14364 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:37.853209 14364 heartbeater.cc:511] Master 127.10.13.62:40523 requested a full tablet report, sending...
I20260925 02:31:37.853609 13902 ts_manager.cc:194] Registered new tserver with Master: ea5f90f4791445ea92955a3b7a198e7f (127.10.13.3:38529)
I20260925 02:31:37.854066 13902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:59439
I20260925 02:31:37.860035 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:37.883256 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.875401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.10.13.1:38187)
0925 02:31:37.875584 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:37.875589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:37.875600 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:37.875627 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:37.875629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:37.875646 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:37.875715 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:37.876280 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.877058 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.877768 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.878084 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.878147 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.880395 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:37.880405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:37.880405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:37.880412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:37.882073 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.882400 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.882405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.882405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.882477 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.882750 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.882751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.882751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.882908 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:37.883027 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:37.883107 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 02:31:37.883173 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:37.891441 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:37.884725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.10.13.62:40523)
0925 02:31:37.884799 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:37.884801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:37.884851 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:31:37.884950 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:37.884951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:37.884960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:37.885006 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:37.885441 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.885922 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.886550 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:37.886733 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:37.886780 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:37.888918 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:37.888920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:37.888920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:37.888928 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:37.890257 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.890584 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.890586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.890586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.890654 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:37.890861 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:37.890862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:37.890862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:37.891014 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:37.891119 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:37.891249 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 02:31:37.891329 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 02:31:37.891733 13902 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34800
I20260925 02:31:37.892165 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 13964
I20260925 02:31:37.897482 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 14098
I20260925 02:31:37.902518 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 14234
I20260925 02:31:37.907195 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 13869
2026-09-25T02:31:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1893 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[14376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[14376](info): set up 2 sockets
Sep 25 02:31:37 dist-test-slave-q8mg krb5kdc[14376](info): commencing operation
krb5kdc: starting...
W20260925 02:31:38.955356 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:39.082134 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45123
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:45123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:39.171190 14389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:39.171517 14389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:39.171564 14389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:39.173760 14389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:39.173832 14389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:39.173859 14389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:39.173882 14389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:39.173915 14389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:39.176738 14389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:45123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:39.177074 14389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:39.177356 14389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:39.182901 14397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:39.182919 14395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:39.182921 14394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:39.183075 14389 server_base.cc:1034] running on GCE node
I20260925 02:31:39.183349 14389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:39.183673 14389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:39.184911 14389 hybrid_clock.cc:648] HybridClock initialized: now 1790303499184885 us; error 37 us; skew 500 ppm
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:39.188510 14389 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:39.188989 14389 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:39.189070 14389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:39.189254 14389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:39.190886 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "450622f5f8e64dab8be94ffb114d5d43"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "df013998cb88ac5e803e3579a4e1dfb2"
server_key_iv: "a2d04c3ccb297179e58725a6a042f866"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.191357 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "450622f5f8e64dab8be94ffb114d5d43"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "df013998cb88ac5e803e3579a4e1dfb2"
server_key_iv: "a2d04c3ccb297179e58725a6a042f866"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.193230 14389 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 02:31:39.194293 14403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.194476 14389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:39.194576 14389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "450622f5f8e64dab8be94ffb114d5d43"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "df013998cb88ac5e803e3579a4e1dfb2"
server_key_iv: "a2d04c3ccb297179e58725a6a042f866"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.194646 14389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:39.207463 14389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:39.209532 14389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:39.209702 14389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:39.213780 14389 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:45123
I20260925 02:31:39.213809 14455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:45123 every 8 connection(s)
I20260925 02:31:39.214124 14389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:39.215124 14456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:39.217362 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 14389
I20260925 02:31:39.217494 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:39.217664 10292 external_mini_cluster.cc:1428] Setting key f52b13b2e1a28674aa141f538ecbf598
I20260925 02:31:39.217708 14456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43: Bootstrap starting.
I20260925 02:31:39.218461 14456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:39.218775 14456 log.cc:824] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:39.219480 14456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43: No bootstrap required, opened a new log
I20260925 02:31:39.221416 14456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450622f5f8e64dab8be94ffb114d5d43" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45123 } }
I20260925 02:31:39.221583 14456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:39.221632 14456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450622f5f8e64dab8be94ffb114d5d43, State: Initialized, Role: FOLLOWER
I20260925 02:31:39.221774 14456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450622f5f8e64dab8be94ffb114d5d43" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45123 } }
I20260925 02:31:39.221858 14456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:39.221902 14456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:39.221951 14456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:39.222486 14456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450622f5f8e64dab8be94ffb114d5d43" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45123 } }
I20260925 02:31:39.222606 14456 leader_election.cc:302] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 450622f5f8e64dab8be94ffb114d5d43; no voters:
I20260925 02:31:39.222800 14456 leader_election.cc:288] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:39.222857 14461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:39.222999 14461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [term 1 LEADER]: Becoming Leader. State: Replica: 450622f5f8e64dab8be94ffb114d5d43, State: Running, Role: LEADER
I20260925 02:31:39.223075 14456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:39.223094 14461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450622f5f8e64dab8be94ffb114d5d43" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45123 } }
I20260925 02:31:39.223371 14461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450622f5f8e64dab8be94ffb114d5d43. Latest consensus state: current_term: 1 leader_uuid: "450622f5f8e64dab8be94ffb114d5d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450622f5f8e64dab8be94ffb114d5d43" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45123 } } }
I20260925 02:31:39.223466 14461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:39.223380 14463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "450622f5f8e64dab8be94ffb114d5d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450622f5f8e64dab8be94ffb114d5d43" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45123 } } }
I20260925 02:31:39.223623 14463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:39.223757 14465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:39.224289 14465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:39.226382 14465 catalog_manager.cc:1347] Generated new cluster ID: 5bc59446139c415dbc3c1c578cced364
I20260925 02:31:39.226441 14465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:39.233292 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.218209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.10.13.62:45123)
0925 02:31:39.218389 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.218399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.218433 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:31:39.218948 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.218956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.219007 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.219155 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:39.219814 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.220687 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.221513 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.222134 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.222257 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.225275 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:39.225279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:39.225279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:39.225292 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:39.230584 (+ 5292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.230982 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.230986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.230986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.231083 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.231303 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.231305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.231305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.231544 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:39.232897 (+ 1353us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.232996 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.233134 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260925 02:31:39.248533 14465 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:39.249330 14465 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:39.255190 14465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43: Generated new TSK 0
I20260925 02:31:39.255406 14465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:39.291874 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45123
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:39.375334 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:39.375584 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:39.375626 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:39.376849 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:39.376945 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:39.376991 14483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:39.378576 14483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:39.378859 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:39.379112 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:39.384204 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:39.384228 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:39.384436 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:39.384446 14483 server_base.cc:1034] running on GCE node
I20260925 02:31:39.384759 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:39.385058 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:39.386231 14483 hybrid_clock.cc:648] HybridClock initialized: now 1790303499386198 us; error 50 us; skew 500 ppm
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:39.388921 14483 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:39.389308 14483 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:39.389382 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:39.389535 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:39.390610 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "a9ac7dc0e43e44a99754230f1f33cc62"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "230326ad5f43c7d42641f0b4437c757b"
server_key_iv: "ad93d212c212e28738ff07dee4e40edb"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.390980 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "a9ac7dc0e43e44a99754230f1f33cc62"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "230326ad5f43c7d42641f0b4437c757b"
server_key_iv: "ad93d212c212e28738ff07dee4e40edb"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.392320 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:39.393091 14497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.393267 14483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:39.393378 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "a9ac7dc0e43e44a99754230f1f33cc62"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "230326ad5f43c7d42641f0b4437c757b"
server_key_iv: "ad93d212c212e28738ff07dee4e40edb"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.393451 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:39.404212 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:39.406193 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:39.406360 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:39.406598 14483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:39.406930 14483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:39.406984 14483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.407018 14483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:39.407050 14483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.412724 14483 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:36961
I20260925 02:31:39.412802 14610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:36961 every 8 connection(s)
I20260925 02:31:39.413097 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:39.417363 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 14483
I20260925 02:31:39.417513 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:39.417721 10292 external_mini_cluster.cc:1428] Setting key 09290c877569edfe0c6bda9e69565f51
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:39.423540 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.413826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:39455 (local address 127.10.13.62:45123)
0925 02:31:39.413882 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.413884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.414112 (+ 228us) server_negotiation.cc:407] Connection header received
0925 02:31:39.414638 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.414639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.414652 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.414707 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:39.415705 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.416223 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.417457 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.417770 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.417853 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.420550 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:39.420551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:39.420551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:39.420559 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:39.422015 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.422453 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.422454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.422455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.422518 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.422774 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.422775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.422776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.422896 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:39.422976 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.423352 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.423449 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:31:39.423830 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.413793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45123 (local address 127.10.13.1:39455)
0925 02:31:39.413945 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:39.413991 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:39.414514 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:39.414779 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:39.414783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:39.415066 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:39.415578 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.415583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.416331 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.416332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.417340 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.417343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.417864 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.417866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.418016 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.418601 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:39.418613 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:39.420433 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:39.422128 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:39.422136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:39.422138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:39.422371 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:39.422589 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:39.422590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:39.422590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:39.422705 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:39.423001 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:39.423008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:39.423201 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 02:31:39.423329 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 02:31:39.423499 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260925 02:31:39.424330 14611 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45123
I20260925 02:31:39.424450 14611 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:39.424623 14611 heartbeater.cc:511] Master 127.10.13.62:45123 requested a full tablet report, sending...
I20260925 02:31:39.425212 14420 ts_manager.cc:194] Registered new tserver with Master: a9ac7dc0e43e44a99754230f1f33cc62 (127.10.13.1:36961)
I20260925 02:31:39.425823 14420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:39455
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:39.444305 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45123
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:39.528172 14616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:39.528389 14616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:39.528414 14616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:39.529666 14616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:39.529712 14616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:39.529740 14616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:39.531234 14616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:39.531433 14616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:39.531615 14616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:39.536695 14622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:39.536715 14621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:39.536706 14624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:39.537062 14616 server_base.cc:1034] running on GCE node
I20260925 02:31:39.537307 14616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:39.537568 14616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:39.538748 14616 hybrid_clock.cc:648] HybridClock initialized: now 1790303499538721 us; error 34 us; skew 500 ppm
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:39.541981 14616 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:39.542289 14616 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:39.542340 14616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:39.542439 14616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:39.543470 14616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "1c9ba704094d47929aae4a5af2cfd976"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "e17fd8eb40873e632b3409650198191a"
server_key_iv: "c69ea3126ca06684d731d42a4e453714"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.543781 14616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "1c9ba704094d47929aae4a5af2cfd976"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "e17fd8eb40873e632b3409650198191a"
server_key_iv: "c69ea3126ca06684d731d42a4e453714"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.545081 14616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 02:31:39.545816 14630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.546010 14616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:39.546113 14616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "1c9ba704094d47929aae4a5af2cfd976"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "e17fd8eb40873e632b3409650198191a"
server_key_iv: "c69ea3126ca06684d731d42a4e453714"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.546201 14616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:39.554139 14616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:39.555924 14616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:39.556083 14616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:39.556311 14616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:39.556638 14616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:39.556699 14616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.556749 14616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:39.556774 14616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.562673 14616 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:37355
I20260925 02:31:39.562748 14743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:37355 every 8 connection(s)
I20260925 02:31:39.563021 14616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:39.569406 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 14616
I20260925 02:31:39.569522 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:39.569712 10292 external_mini_cluster.cc:1428] Setting key cb55f2c16aad1449011e234f2bb23330
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:39.573673 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.563778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:47851 (local address 127.10.13.62:45123)
0925 02:31:39.563853 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.563855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.564063 (+ 208us) server_negotiation.cc:407] Connection header received
0925 02:31:39.564532 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.564533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.564543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.564617 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:39.565859 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.566375 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.567465 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.567663 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.567720 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.570450 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:39.570452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:39.570452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:39.570460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:39.572205 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.572624 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.572626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.572626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.572691 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.573004 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.573005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.573006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.573127 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:39.573203 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.573489 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.573582 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:31:39.573869 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.563775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45123 (local address 127.10.13.2:47851)
0925 02:31:39.563939 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:39.563976 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:39.564448 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:39.564761 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:39.564766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:39.565181 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:39.565762 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.565768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.566471 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.566472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.567369 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.567371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.567748 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.567749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.567860 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.568552 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:39.568562 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:39.570333 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:39.572304 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:39.572319 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:39.572321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:39.572541 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:39.572758 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:39.572758 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:39.572759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:39.572923 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:39.573195 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:39.573202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:39.573346 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 02:31:39.573451 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:31:39.573620 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260925 02:31:39.574402 14744 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45123
I20260925 02:31:39.574544 14744 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:39.574745 14744 heartbeater.cc:511] Master 127.10.13.62:45123 requested a full tablet report, sending...
I20260925 02:31:39.575141 14420 ts_manager.cc:194] Registered new tserver with Master: 1c9ba704094d47929aae4a5af2cfd976 (127.10.13.2:37355)
I20260925 02:31:39.575524 14420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:47851
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:39.597833 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45123
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:39.681205 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:39.681707 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:39.681806 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:39.683070 14749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:39.683139 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:39.683187 14749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:39.684787 14749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:39.685091 14749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:39.685333 14749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:39.690371 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:39.690372 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:39.690387 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:39.690614 14749 server_base.cc:1034] running on GCE node
I20260925 02:31:39.690874 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:39.691103 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:39.692296 14749 hybrid_clock.cc:648] HybridClock initialized: now 1790303499692287 us; error 41 us; skew 500 ppm
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:39.695010 14749 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:39.695377 14749 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:39.695465 14749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:39.695614 14749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:39.696697 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "36909e11f91f46b1b425e7f87bf76399"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "336839b6fd3b29b39aefe12ad38e234b"
server_key_iv: "43d918dc0ac02f15924cfa2702d45606"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.697091 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "36909e11f91f46b1b425e7f87bf76399"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "336839b6fd3b29b39aefe12ad38e234b"
server_key_iv: "43d918dc0ac02f15924cfa2702d45606"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.698470 14749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:39.699271 14763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.699450 14749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:39.699560 14749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "36909e11f91f46b1b425e7f87bf76399"
format_stamp: "Formatted at 2026-09-25 02:31:39 on dist-test-slave-q8mg"
server_key: "336839b6fd3b29b39aefe12ad38e234b"
server_key_iv: "43d918dc0ac02f15924cfa2702d45606"
server_key_version: "encryptionkey@0"
I20260925 02:31:39.699637 14749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:39.719586 14749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:39.722031 14749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:39.722199 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:39.722452 14749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:39.722849 14749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:39.722913 14749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.722947 14749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:39.722975 14749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:39.729681 14749 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:42817
I20260925 02:31:39.729717 14877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:42817 every 8 connection(s)
I20260925 02:31:39.729990 14749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:39.733057 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 14749
I20260925 02:31:39.733155 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:39.733332 10292 external_mini_cluster.cc:1428] Setting key 1942139cd7110399b0c5cb00f9a40961
Sep 25 02:31:39 dist-test-slave-q8mg krb5kdc[14376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:39.739403 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.730694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:52991 (local address 127.10.13.62:45123)
0925 02:31:39.730744 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.730746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.730921 (+ 175us) server_negotiation.cc:407] Connection header received
0925 02:31:39.731396 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.731397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.731408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.731462 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:39.732398 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.732965 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.734438 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.734646 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.734723 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.736804 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:39.736806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:39.736806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:39.736814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:39.737993 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.738428 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.738429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.738430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.738492 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.738757 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.738758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.738758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.738874 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:39.738953 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.739229 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.739310 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:31:39.739609 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.730685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45123 (local address 127.10.13.3:52991)
0925 02:31:39.730812 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:39.730848 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:39.731309 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:39.731515 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:39.731519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:39.731768 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:39.732305 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.732311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.733060 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.733062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.734349 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.734352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.734731 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.734732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.734870 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.735334 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:39.735343 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:39.736703 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:39.738115 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:39.738123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:39.738125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:39.738348 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:39.738573 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:39.738573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:39.738574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:39.738683 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:39.738943 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:39.738950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:39.739094 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 02:31:39.739208 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:31:39.739365 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260925 02:31:39.740093 14878 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45123
I20260925 02:31:39.740180 14878 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:39.740302 14878 heartbeater.cc:511] Master 127.10.13.62:45123 requested a full tablet report, sending...
I20260925 02:31:39.740629 14420 ts_manager.cc:194] Registered new tserver with Master: 36909e11f91f46b1b425e7f87bf76399 (127.10.13.3:42817)
I20260925 02:31:39.741130 14420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:52991
I20260925 02:31:39.745330 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:31:39.752264 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.746555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.10.13.62:45123)
0925 02:31:39.746630 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.746632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.746711 (+ 79us) server_negotiation.cc:407] Connection header received
0925 02:31:39.746751 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.746751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.746760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.746805 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:39.747296 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.747761 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.748457 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.748630 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.748680 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.749919 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:39.749970 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:39.749970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:39.749977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:39.751250 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.751548 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.751549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.751549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.751601 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:39.751764 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:39.751764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:39.751764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:39.751901 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:39.751977 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.752087 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.752149 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:31:39.754014 14420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:39.755820 14420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:39.762501 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.758169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:42817 (local address 127.0.0.1:59632)
0925 02:31:39.758576 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:39.758587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:39.758643 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:39.758848 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:39.758851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:39.758853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:39.758925 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.758927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.760267 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.760267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.761475 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.761477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.762159 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.762160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.762248 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.762262 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:31:39.762339 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:31:39.762395 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":9,"thread_start_us":89,"threads_started":1}
I20260925 02:31:39.762595 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.758421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.10.13.2:37355)
0925 02:31:39.758982 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.758987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.758996 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:39.759032 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.759034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.759050 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.759125 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:39.759348 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.760020 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.761561 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.762271 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.762339 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.762392 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.762476 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.762518 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":68,"threads_started":1}
I20260925 02:31:39.762681 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.758403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.10.13.3:42817)
0925 02:31:39.758576 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.758581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.758659 (+ 78us) server_negotiation.cc:407] Connection header received
0925 02:31:39.758708 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.758711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.758730 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.758821 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:39.759017 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.760154 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.761576 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.762050 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.762159 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.762222 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.762540 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.762589 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260925 02:31:39.762702 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.758351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.10.13.1:36961)
0925 02:31:39.758700 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.758704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.758713 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:39.758749 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.758751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.758767 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.758840 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:39.759082 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.760182 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.761378 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.762089 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.762161 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.762221 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.762569 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.762615 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":117,"threads_started":1}
I20260925 02:31:39.763062 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.758184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:37355 (local address 127.0.0.1:49624)
0925 02:31:39.758577 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:39.758587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:39.758637 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:39.759215 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:39.759221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:39.759224 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:39.759262 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.759264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.760121 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.760122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.761472 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.761475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.762349 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.762349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.762400 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.762403 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:39.762451 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:31:39.762488 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":32,"thread_start_us":79,"threads_started":1}
I20260925 02:31:39.762521 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.758164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:36961 (local address 127.0.0.1:39900)
0925 02:31:39.758314 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:39.758357 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:39.758476 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:39.758947 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:39.758948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:39.758950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:39.758986 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.758987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.760290 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.760291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.761285 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.761288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.762259 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.762261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.762330 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.762334 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:39.762382 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:31:39.762425 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260925 02:31:39.763803 14812 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9094894fab40209f3c071a269d3812 (DEFAULT_TABLE table=test-table [id=1e185c93c7004c288444bf6699a6213d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:39.763954 14545 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9094894fab40209f3c071a269d3812 (DEFAULT_TABLE table=test-table [id=1e185c93c7004c288444bf6699a6213d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:39.763953 14678 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9094894fab40209f3c071a269d3812 (DEFAULT_TABLE table=test-table [id=1e185c93c7004c288444bf6699a6213d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:39.764206 14812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9094894fab40209f3c071a269d3812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:39.764287 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9094894fab40209f3c071a269d3812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:39.764287 14678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9094894fab40209f3c071a269d3812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:39.767083 14895 tablet_bootstrap.cc:492] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Bootstrap starting.
I20260925 02:31:39.767623 14897 tablet_bootstrap.cc:492] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Bootstrap starting.
I20260925 02:31:39.767942 14895 tablet_bootstrap.cc:654] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:39.768294 14897 tablet_bootstrap.cc:654] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:39.768589 14895 log.cc:824] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:39.768949 14897 log.cc:824] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:39.769317 14896 tablet_bootstrap.cc:492] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62: Bootstrap starting.
I20260925 02:31:39.769518 14895 tablet_bootstrap.cc:492] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: No bootstrap required, opened a new log
I20260925 02:31:39.769601 14895 ts_tablet_manager.cc:1397] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:31:39.769851 14897 tablet_bootstrap.cc:492] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: No bootstrap required, opened a new log
I20260925 02:31:39.769950 14897 ts_tablet_manager.cc:1397] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:31:39.769970 14896 tablet_bootstrap.cc:654] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:39.770563 14896 log.cc:824] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:39.771363 14896 tablet_bootstrap.cc:492] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62: No bootstrap required, opened a new log
I20260925 02:31:39.771435 14896 ts_tablet_manager.cc:1397] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:31:39.771683 14897 raft_consensus.cc:348] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.771847 14897 raft_consensus.cc:374] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:39.771893 14897 raft_consensus.cc:729] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9ba704094d47929aae4a5af2cfd976, State: Initialized, Role: FOLLOWER
I20260925 02:31:39.771798 14895 raft_consensus.cc:348] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.771951 14895 raft_consensus.cc:374] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:39.771993 14895 raft_consensus.cc:729] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36909e11f91f46b1b425e7f87bf76399, State: Initialized, Role: FOLLOWER
I20260925 02:31:39.772044 14897 consensus_queue.cc:260] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.772125 14895 consensus_queue.cc:260] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.772374 14897 ts_tablet_manager.cc:1428] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 02:31:39.772395 14895 ts_tablet_manager.cc:1428] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:39.772410 14878 heartbeater.cc:503] Master 127.10.13.62:45123 was elected leader, sending a full tablet report...
I20260925 02:31:39.772501 14744 heartbeater.cc:503] Master 127.10.13.62:45123 was elected leader, sending a full tablet report...
I20260925 02:31:39.773798 14896 raft_consensus.cc:348] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.773999 14896 raft_consensus.cc:374] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:39.774049 14896 raft_consensus.cc:729] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ac7dc0e43e44a99754230f1f33cc62, State: Initialized, Role: FOLLOWER
I20260925 02:31:39.774170 14896 consensus_queue.cc:260] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.774427 14896 ts_tablet_manager.cc:1428] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:39.774477 14611 heartbeater.cc:503] Master 127.10.13.62:45123 was elected leader, sending a full tablet report...
W20260925 02:31:39.813632 14745 tablet.cc:2369] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:31:39.914031 14612 tablet.cc:2369] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:39.930595 14901 raft_consensus.cc:482] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:39.930773 14901 raft_consensus.cc:504] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.931201 14901 leader_election.cc:288] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ac7dc0e43e44a99754230f1f33cc62 (127.10.13.1:36961), 36909e11f91f46b1b425e7f87bf76399 (127.10.13.3:42817)
I20260925 02:31:39.935251 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.931527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:55791 (local address 127.10.13.1:36961)
0925 02:31:39.931581 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.931583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.931712 (+ 129us) server_negotiation.cc:407] Connection header received
0925 02:31:39.931742 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.931743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.931757 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.931809 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:39.931978 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.932746 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.934339 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.934918 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.934996 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.935007 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.935137 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.935182 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:31:39.935259 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.931425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:36961 (local address 127.10.13.2:55791)
0925 02:31:39.931620 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:39.931647 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:39.931703 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:39.931826 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:39.931826 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:39.931827 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:39.931901 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.931903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.932831 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.932831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.934242 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.934245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.935003 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.935004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.935058 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.935062 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:39.935103 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 02:31:39.935167 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":91,"thread_start_us":116,"threads_started":1}
I20260925 02:31:39.935418 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.931425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:42817 (local address 127.10.13.2:57319)
0925 02:31:39.931490 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:39.931505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:39.931578 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:39.931815 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:39.931816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:39.931818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:39.931901 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.931903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.932830 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.932831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.934242 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:39.934245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.935180 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.935181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:39.935247 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.935251 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:39.935305 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:31:39.935340 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":87}
I20260925 02:31:39.935556 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.931580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:57319 (local address 127.10.13.3:42817)
0925 02:31:39.931637 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.931639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.931648 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:39.931682 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.931699 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.931713 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.931776 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:39.932037 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.932745 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.934495 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.935071 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.935159 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.935169 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.935460 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.935499 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:31:39.935812 14565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9094894fab40209f3c071a269d3812" candidate_uuid: "1c9ba704094d47929aae4a5af2cfd976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" is_pre_election: true
I20260925 02:31:39.935845 14832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9094894fab40209f3c071a269d3812" candidate_uuid: "1c9ba704094d47929aae4a5af2cfd976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36909e11f91f46b1b425e7f87bf76399" is_pre_election: true
I20260925 02:31:39.935979 14832 raft_consensus.cc:2454] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9ba704094d47929aae4a5af2cfd976 in term 0.
I20260925 02:31:39.935979 14565 raft_consensus.cc:2454] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9ba704094d47929aae4a5af2cfd976 in term 0.
I20260925 02:31:39.936231 14632 leader_election.cc:302] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9ba704094d47929aae4a5af2cfd976, a9ac7dc0e43e44a99754230f1f33cc62; no voters:
I20260925 02:31:39.936398 14901 raft_consensus.cc:2781] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:39.936472 14901 raft_consensus.cc:482] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:39.936491 14901 raft_consensus.cc:3037] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:39.937234 14901 raft_consensus.cc:504] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.937389 14901 leader_election.cc:288] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [CANDIDATE]: Term 1 election: Requested vote from peers a9ac7dc0e43e44a99754230f1f33cc62 (127.10.13.1:36961), 36909e11f91f46b1b425e7f87bf76399 (127.10.13.3:42817)
I20260925 02:31:39.937559 14832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9094894fab40209f3c071a269d3812" candidate_uuid: "1c9ba704094d47929aae4a5af2cfd976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36909e11f91f46b1b425e7f87bf76399"
I20260925 02:31:39.937623 14832 raft_consensus.cc:3037] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:39.938303 14832 raft_consensus.cc:2454] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9ba704094d47929aae4a5af2cfd976 in term 1.
I20260925 02:31:39.938428 14565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9094894fab40209f3c071a269d3812" candidate_uuid: "1c9ba704094d47929aae4a5af2cfd976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ac7dc0e43e44a99754230f1f33cc62"
I20260925 02:31:39.938514 14634 leader_election.cc:302] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9ba704094d47929aae4a5af2cfd976, 36909e11f91f46b1b425e7f87bf76399; no voters:
I20260925 02:31:39.938514 14565 raft_consensus.cc:3037] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:39.938627 14901 raft_consensus.cc:2781] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:39.938773 14901 raft_consensus.cc:686] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 1 LEADER]: Becoming Leader. State: Replica: 1c9ba704094d47929aae4a5af2cfd976, State: Running, Role: LEADER
I20260925 02:31:39.938869 14901 consensus_queue.cc:237] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } }
I20260925 02:31:39.939432 14565 raft_consensus.cc:2454] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9ba704094d47929aae4a5af2cfd976 in term 1.
I20260925 02:31:39.939795 14420 catalog_manager.cc:5495] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c9ba704094d47929aae4a5af2cfd976 (127.10.13.2). New cstate: current_term: 1 leader_uuid: "1c9ba704094d47929aae4a5af2cfd976" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c9ba704094d47929aae4a5af2cfd976" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 37355 } health_report { overall_health: HEALTHY } } }
W20260925 02:31:39.980571 14879 tablet.cc:2369] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:39.988152 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:39.985975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.10.13.2:37355)
0925 02:31:39.986037 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:39.986039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:39.986047 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:39.986107 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:39.986108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:39.986129 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:39.986193 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:39.986306 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.986765 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.987473 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:39.987655 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:39.987703 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:39.987845 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:39.987938 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:39.987997 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:39.988034 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 02:31:39.988049 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:31:39.990131 14832 raft_consensus.cc:1261] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 1 FOLLOWER]: Refusing update from remote peer 1c9ba704094d47929aae4a5af2cfd976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:39.990151 14565 raft_consensus.cc:1261] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 1 FOLLOWER]: Refusing update from remote peer 1c9ba704094d47929aae4a5af2cfd976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:39.990479 14901 consensus_queue.cc:1035] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ac7dc0e43e44a99754230f1f33cc62" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:39.990643 14905 consensus_queue.cc:1035] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36909e11f91f46b1b425e7f87bf76399" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:39.992928 14908 mvcc.cc:205] Tried to move back new op lower bound from 7333083135956316160 to 7333083135750017024. Current Snapshot: MvccSnapshot[applied={T|T < 7333083135956316160}]
I20260925 02:31:39.993476 14909 mvcc.cc:205] Tried to move back new op lower bound from 7333083135956316160 to 7333083135750017024. Current Snapshot: MvccSnapshot[applied={T|T < 7333083135956316160 or (T in {7333083135956316160})}]
I20260925 02:31:39.993809 14910 mvcc.cc:205] Tried to move back new op lower bound from 7333083135956316160 to 7333083135750017024. Current Snapshot: MvccSnapshot[applied={T|T < 7333083135956316160 or (T in {7333083135956316160})}]
I20260925 02:31:39.996050 14419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:39.996157 14419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:39.996956 14419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 450622f5f8e64dab8be94ffb114d5d43: Sending DeleteTablet for 3 replicas of tablet 4d9094894fab40209f3c071a269d3812
I20260925 02:31:39.997291 14545 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9094894fab40209f3c071a269d3812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:39 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39900
I20260925 02:31:39.997364 14678 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9094894fab40209f3c071a269d3812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:39 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:49624
I20260925 02:31:39.997383 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9094894fab40209f3c071a269d3812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:39 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:59632
I20260925 02:31:39.997617 14919 tablet_replica.cc:330] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: stopping tablet replica
I20260925 02:31:39.997726 14919 raft_consensus.cc:2229] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:39.997793 14918 tablet_replica.cc:330] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62: stopping tablet replica
I20260925 02:31:39.997835 14919 raft_consensus.cc:2258] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:39.997895 14918 raft_consensus.cc:2229] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:39.997967 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 14483
I20260925 02:31:39.998003 14918 raft_consensus.cc:2258] T 4d9094894fab40209f3c071a269d3812 P a9ac7dc0e43e44a99754230f1f33cc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:39.998361 14919 ts_tablet_manager.cc:1905] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:40.000036 14920 tablet_replica.cc:330] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: stopping tablet replica
I20260925 02:31:40.000133 14920 raft_consensus.cc:2229] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:40.000263 14920 raft_consensus.cc:2258] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:40.000451 14919 ts_tablet_manager.cc:1918] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:40.000535 14919 log.cc:1196] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/4d9094894fab40209f3c071a269d3812
I20260925 02:31:40.000824 14919 ts_tablet_manager.cc:1939] T 4d9094894fab40209f3c071a269d3812 P 36909e11f91f46b1b425e7f87bf76399: Deleting consensus metadata
I20260925 02:31:40.001309 14407 catalog_manager.cc:4841] TS 36909e11f91f46b1b425e7f87bf76399 (127.10.13.3:42817): tablet 4d9094894fab40209f3c071a269d3812 (table test-table [id=1e185c93c7004c288444bf6699a6213d]) successfully deleted
I20260925 02:31:40.001749 14920 ts_tablet_manager.cc:1905] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:40.003793 14920 ts_tablet_manager.cc:1918] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:40.003872 14920 log.cc:1196] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/4d9094894fab40209f3c071a269d3812
I20260925 02:31:40.004139 14920 ts_tablet_manager.cc:1939] T 4d9094894fab40209f3c071a269d3812 P 1c9ba704094d47929aae4a5af2cfd976: Deleting consensus metadata
I20260925 02:31:40.004995 14406 catalog_manager.cc:4841] TS 1c9ba704094d47929aae4a5af2cfd976 (127.10.13.2:37355): tablet 4d9094894fab40209f3c071a269d3812 (table test-table [id=1e185c93c7004c288444bf6699a6213d]) successfully deleted
W20260925 02:31:40.005561 14405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:36961 (error 108)
W20260925 02:31:40.006130 14890 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:40.005970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:36961 (local address 127.0.0.1:39914)
0925 02:31:40.006033 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.006089 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:36961: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:31:40.006258 14405 catalog_manager.cc:4584] TS a9ac7dc0e43e44a99754230f1f33cc62 (127.10.13.1:36961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d9094894fab40209f3c071a269d3812: Network error: Client connection negotiation failed: client connection to 127.10.13.1:36961: connect: Connection reset by peer (error 104)
I20260925 02:31:40.006325 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 14616
I20260925 02:31:40.011132 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 14749
I20260925 02:31:40.015352 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 14389
2026-09-25T02:31:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2107 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T02:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:40Z Disabled control of system clock
I20260925 02:31:40.042611 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39345
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:31:40.124508 14925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:40.124740 14925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:40.124763 14925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:40.125960 14925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:40.126005 14925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:40.126019 14925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:40.126030 14925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:40.126042 14925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:40.127327 14925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:40.127540 14925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:40.127799 14925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:40.132831 14930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.132861 14933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:40.133047 14925 server_base.cc:1034] running on GCE node
W20260925 02:31:40.132884 14931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:40.133381 14925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:40.133702 14925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:40.134879 14925 hybrid_clock.cc:648] HybridClock initialized: now 1790303500134868 us; error 51 us; skew 500 ppm
I20260925 02:31:40.136206 14925 webserver.cc:460] Webserver started at http://127.10.13.62:33163/ using document root <none> and password file <none>
I20260925 02:31:40.136495 14925 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:40.136572 14925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:40.136732 14925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:40.137974 14925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "20d08355be1d47e69233086fb83e4528"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "0bdd6031627ea9b804111fade326be71"
server_key_iv: "5b420240c8e557feddddb6164ccdbcf4"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.138355 14925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "20d08355be1d47e69233086fb83e4528"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "0bdd6031627ea9b804111fade326be71"
server_key_iv: "5b420240c8e557feddddb6164ccdbcf4"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.139761 14925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:40.140578 14939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.140784 14925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:40.140913 14925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "20d08355be1d47e69233086fb83e4528"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "0bdd6031627ea9b804111fade326be71"
server_key_iv: "5b420240c8e557feddddb6164ccdbcf4"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.141006 14925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:40.144412 14925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:40.144711 14925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:40.144901 14925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:40.149000 14925 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:39345
I20260925 02:31:40.149070 14991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:39345 every 8 connection(s)
I20260925 02:31:40.149413 14925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:40.150098 14992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:40.152992 14992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528: Bootstrap starting.
I20260925 02:31:40.153554 14992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:40.153798 14992 log.cc:824] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:40.154508 14992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528: No bootstrap required, opened a new log
I20260925 02:31:40.156082 14992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d08355be1d47e69233086fb83e4528" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39345 } }
I20260925 02:31:40.156222 14992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:40.156244 14992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d08355be1d47e69233086fb83e4528, State: Initialized, Role: FOLLOWER
I20260925 02:31:40.156352 14992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d08355be1d47e69233086fb83e4528" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39345 } }
I20260925 02:31:40.156404 14992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:40.156423 14992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:40.156450 14992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:40.156939 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 14925
I20260925 02:31:40.157035 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:40.157053 14992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d08355be1d47e69233086fb83e4528" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39345 } }
I20260925 02:31:40.157181 14992 leader_election.cc:302] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20d08355be1d47e69233086fb83e4528; no voters:
I20260925 02:31:40.157234 10292 external_mini_cluster.cc:1428] Setting key 21f74a1b485483922e3b3587c90c945b
I20260925 02:31:40.157374 14992 leader_election.cc:288] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:40.157405 14995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:40.157570 14995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [term 1 LEADER]: Becoming Leader. State: Replica: 20d08355be1d47e69233086fb83e4528, State: Running, Role: LEADER
I20260925 02:31:40.157658 14995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d08355be1d47e69233086fb83e4528" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39345 } }
I20260925 02:31:40.157670 14992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:40.158000 14995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20d08355be1d47e69233086fb83e4528. Latest consensus state: current_term: 1 leader_uuid: "20d08355be1d47e69233086fb83e4528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d08355be1d47e69233086fb83e4528" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39345 } } }
I20260925 02:31:40.158089 14995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:40.158021 14996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20d08355be1d47e69233086fb83e4528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d08355be1d47e69233086fb83e4528" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39345 } } }
I20260925 02:31:40.158253 14996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:40.159478 15001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:40.160045 15001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:40.160472 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.158062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.10.13.62:39345)
0925 02:31:40.158418 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.158427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.158460 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:31:40.158593 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.158601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.158652 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.159084 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.159791 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.159794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.159795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.159800 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.159845 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.159860 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.160201 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.160233 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.160291 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":71,"threads_started":1}
I20260925 02:31:40.161755 15001 catalog_manager.cc:1347] Generated new cluster ID: 0f35682c20f248a2bc31ac562b8e0fd2
I20260925 02:31:40.161829 15001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:40.175192 15001 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:40.175414 15001 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:40.183048 15001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528: Generated new TSK 0
I20260925 02:31:40.183274 15001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:31:40.185757 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39345
--builtin_ntp_servers=127.10.13.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:31:40.269100 15016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:40.269347 15016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:40.269385 15016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:40.270618 15016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:40.270684 15016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:40.270728 15016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:40.272132 15016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:40.272413 15016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:40.272733 15016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:40.279170 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:40.279371 15016 server_base.cc:1034] running on GCE node
W20260925 02:31:40.279208 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.279201 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:40.279716 15016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:40.279968 15016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:40.281138 15016 hybrid_clock.cc:648] HybridClock initialized: now 1790303500281122 us; error 31 us; skew 500 ppm
I20260925 02:31:40.282692 15016 webserver.cc:460] Webserver started at http://127.10.13.1:37457/ using document root <none> and password file <none>
I20260925 02:31:40.283008 15016 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:40.283092 15016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:40.283249 15016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:40.284373 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "0e4f30075edf4677b9126f561ed28941"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "f79f194cde897e86661f6eb1206e9a43"
server_key_iv: "eb2f8675902c1481901d59700265b6d4"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.284744 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "0e4f30075edf4677b9126f561ed28941"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "f79f194cde897e86661f6eb1206e9a43"
server_key_iv: "eb2f8675902c1481901d59700265b6d4"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.286331 15016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:40.287271 15030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.287500 15016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:40.287608 15016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "0e4f30075edf4677b9126f561ed28941"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "f79f194cde897e86661f6eb1206e9a43"
server_key_iv: "eb2f8675902c1481901d59700265b6d4"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.287694 15016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:40.291189 15016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:40.291463 15016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:40.291610 15016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:40.291824 15016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:40.292135 15016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:40.292187 15016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.292227 15016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:40.292255 15016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.298128 15016 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:36975
I20260925 02:31:40.298182 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:36975 every 8 connection(s)
I20260925 02:31:40.298560 15016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:40.300721 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15016
I20260925 02:31:40.300845 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:40.301046 10292 external_mini_cluster.cc:1428] Setting key ddb53366f4a354ac4c35449b0a44b069
I20260925 02:31:40.301929 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.299415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:49675 (local address 127.10.13.62:39345)
0925 02:31:40.299502 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.299504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.299753 (+ 249us) server_negotiation.cc:407] Connection header received
0925 02:31:40.300215 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.300216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.300222 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.300289 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.301672 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.301674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.301674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.301691 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.301710 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.301715 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.301765 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.301843 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.301857 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:31:40.302121 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39345
--builtin_ntp_servers=127.10.13.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:31:40.302148 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.299368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39345 (local address 127.10.13.1:49675)
0925 02:31:40.299616 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.299659 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.300133 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.300361 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.300364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.300373 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:40.301507 (+ 1134us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:31:40.301508 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:40.301545 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:31:40.301576 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:40.301774 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:40.301779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:40.301783 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:40.301856 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:31:40.301920 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":125,"threads_started":1}
I20260925 02:31:40.303755 15144 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39345
I20260925 02:31:40.303840 15144 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:40.303999 15144 heartbeater.cc:511] Master 127.10.13.62:39345 requested a full tablet report, sending...
I20260925 02:31:40.304586 14956 ts_manager.cc:194] Registered new tserver with Master: 0e4f30075edf4677b9126f561ed28941 (127.10.13.1:36975)
W20260925 02:31:40.386370 15147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:40.386611 15147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:40.386649 15147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:40.387846 15147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:40.387916 15147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:40.387961 15147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:40.389349 15147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:40.389629 15147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:40.389933 15147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:40.395504 15155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.395648 15152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.395514 15153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:40.395675 15147 server_base.cc:1034] running on GCE node
I20260925 02:31:40.396013 15147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:40.396293 15147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:40.397459 15147 hybrid_clock.cc:648] HybridClock initialized: now 1790303500397453 us; error 37 us; skew 500 ppm
I20260925 02:31:40.398705 15147 webserver.cc:460] Webserver started at http://127.10.13.2:33739/ using document root <none> and password file <none>
I20260925 02:31:40.398960 15147 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:40.399030 15147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:40.399216 15147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:40.400311 15147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "a2fe554c6fe64b3eb56822329bef8d36"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e0e4daee964afc597d4b8fe1d86b2901"
server_key_iv: "898f3285bca23723d1391d7c8ec2a0a8"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.400688 15147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "a2fe554c6fe64b3eb56822329bef8d36"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e0e4daee964afc597d4b8fe1d86b2901"
server_key_iv: "898f3285bca23723d1391d7c8ec2a0a8"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.402182 15147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:40.403062 15161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.403244 15147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:40.403347 15147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "a2fe554c6fe64b3eb56822329bef8d36"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e0e4daee964afc597d4b8fe1d86b2901"
server_key_iv: "898f3285bca23723d1391d7c8ec2a0a8"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.403436 15147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:40.406896 15147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:40.407222 15147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:40.407374 15147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:40.407596 15147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:40.407915 15147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:40.407970 15147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.408010 15147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:40.408038 15147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.413975 15147 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:44409
I20260925 02:31:40.414014 15274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:44409 every 8 connection(s)
I20260925 02:31:40.414368 15147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:40.417320 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15147
I20260925 02:31:40.417428 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:40.417603 10292 external_mini_cluster.cc:1428] Setting key cacef0c4bc60d6735761a5cbf241032b
I20260925 02:31:40.417631 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.415134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:34785 (local address 127.10.13.62:39345)
0925 02:31:40.415197 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.415199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.415457 (+ 258us) server_negotiation.cc:407] Connection header received
0925 02:31:40.415983 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.415984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.415992 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.416043 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.417354 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.417355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.417356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.417361 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.417379 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.417383 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.417434 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.417545 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.417561 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:31:40.417853 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.415134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39345 (local address 127.10.13.2:34785)
0925 02:31:40.415340 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.415374 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.415863 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.416121 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.416124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.416132 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:40.417191 (+ 1059us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:31:40.417192 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:40.417221 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:31:40.417251 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:40.417460 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:40.417466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:40.417470 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:40.417542 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:31:40.417607 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260925 02:31:40.418439 15275 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39345
I20260925 02:31:40.418541 15275 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:40.418708 15275 heartbeater.cc:511] Master 127.10.13.62:39345 requested a full tablet report, sending...
I20260925 02:31:40.418722 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39345
--builtin_ntp_servers=127.10.13.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:31:40.419168 14955 ts_manager.cc:194] Registered new tserver with Master: a2fe554c6fe64b3eb56822329bef8d36 (127.10.13.2:44409)
W20260925 02:31:40.501956 15278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:40.502220 15278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:40.502261 15278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:40.503573 15278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:40.503641 15278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:40.503684 15278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:40.505064 15278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:40.505373 15278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:40.505681 15278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:40.510939 15286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.510931 15283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.510927 15284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:40.511710 15278 server_base.cc:1034] running on GCE node
I20260925 02:31:40.511936 15278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:40.512187 15278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:40.513365 15278 hybrid_clock.cc:648] HybridClock initialized: now 1790303500513345 us; error 30 us; skew 500 ppm
I20260925 02:31:40.514675 15278 webserver.cc:460] Webserver started at http://127.10.13.3:41283/ using document root <none> and password file <none>
I20260925 02:31:40.514932 15278 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:40.515004 15278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:40.515173 15278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:40.516326 15278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "959afa8883b2475b9f576ea7c85f8622"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "7e16400a00542f109211b592731aa026"
server_key_iv: "e9ad464896d0e5a95d7f707c1e598eff"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.516734 15278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "959afa8883b2475b9f576ea7c85f8622"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "7e16400a00542f109211b592731aa026"
server_key_iv: "e9ad464896d0e5a95d7f707c1e598eff"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.518090 15278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:40.518826 15292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.518999 15278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:40.519101 15278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "959afa8883b2475b9f576ea7c85f8622"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "7e16400a00542f109211b592731aa026"
server_key_iv: "e9ad464896d0e5a95d7f707c1e598eff"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.519184 15278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:40.522621 15278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:40.522889 15278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:40.523032 15278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:40.523247 15278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:40.523568 15278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:40.523630 15278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.523669 15278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:40.523697 15278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.530014 15278 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:41413
I20260925 02:31:40.530058 15405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:41413 every 8 connection(s)
I20260925 02:31:40.530385 15278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:40.533547 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.531068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:53439 (local address 127.10.13.62:39345)
0925 02:31:40.531117 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.531119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.531349 (+ 230us) server_negotiation.cc:407] Connection header received
0925 02:31:40.531867 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.531869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.531877 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.531929 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.533246 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.533248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.533248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.533254 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.533272 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.533276 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.533357 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.533454 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.533474 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:31:40.533752 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.531072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39345 (local address 127.10.13.3:53439)
0925 02:31:40.531228 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.531261 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.531750 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.532003 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.532006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.532014 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:40.533077 (+ 1063us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:31:40.533078 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:40.533110 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:31:40.533138 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:40.533357 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:40.533364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:40.533368 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:40.533464 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:31:40.533528 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260925 02:31:40.533877 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15278
I20260925 02:31:40.533996 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:40.534212 10292 external_mini_cluster.cc:1428] Setting key 543c6a202a7e053ab83b9fb859308a0c
I20260925 02:31:40.534381 15406 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39345
I20260925 02:31:40.534484 15406 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:40.534657 15406 heartbeater.cc:511] Master 127.10.13.62:39345 requested a full tablet report, sending...
I20260925 02:31:40.535032 14956 ts_manager.cc:194] Registered new tserver with Master: 959afa8883b2475b9f576ea7c85f8622 (127.10.13.3:41413)
I20260925 02:31:40.546242 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:31:40.549014 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.547807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.10.13.62:39345)
0925 02:31:40.547905 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.547907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.548013 (+ 106us) server_negotiation.cc:407] Connection header received
0925 02:31:40.548111 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.548111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.548120 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.548200 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.548760 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.548762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.548763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.548767 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.548782 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.548786 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.548836 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.548914 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.548932 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:31:40.550657 14956 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:40.552147 14956 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:40.556262 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.554528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:41413 (local address 127.0.0.1:33620)
0925 02:31:40.555082 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.555092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.555155 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.555443 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.555444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.555447 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:40.555866 (+ 419us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:31:40.555866 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:40.555869 (+ 3us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:31:40.555875 (+ 6us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:40.556099 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:40.556106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:40.556109 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:40.556170 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 02:31:40.556185 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260925 02:31:40.556406 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.554528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:36975 (local address 127.0.0.1:35752)
0925 02:31:40.554894 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.554937 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.554999 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.555172 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.555179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.555187 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:40.555731 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:31:40.555732 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:40.555750 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:31:40.555775 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:40.556260 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:40.556261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:40.556262 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 02:31:40.556312 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 02:31:40.556326 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":111,"threads_started":1}
I20260925 02:31:40.556517 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.554684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.10.13.2:44409)
0925 02:31:40.554864 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.554871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.555164 (+ 293us) server_negotiation.cc:407] Connection header received
0925 02:31:40.555263 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.555267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.555278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.555327 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.556163 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.556172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.556173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.556180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.556213 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.556240 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.556298 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.556394 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.556412 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260925 02:31:40.556654 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.554784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.10.13.3:41413)
0925 02:31:40.555283 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.555288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.555298 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:40.555324 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.555326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.555337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.555441 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.555958 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.555967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.555967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.555971 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.556009 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.556027 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.556092 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.556531 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.556549 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":90,"threads_started":1}
I20260925 02:31:40.556686 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.554833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.10.13.1:36975)
0925 02:31:40.554997 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.555001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.555012 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:40.555062 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.555065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.555077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.555169 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.556124 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.556127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.556128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.556132 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.556173 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.556189 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.556251 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.556567 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.556586 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260925 02:31:40.556692 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.554582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:44409 (local address 127.0.0.1:56402)
0925 02:31:40.555102 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.555112 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.555188 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.555628 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.555629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.555632 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:40.556027 (+ 395us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:31:40.556028 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:40.556036 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:31:40.556050 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:40.556303 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:40.556304 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:40.556304 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 02:31:40.556624 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 02:31:40.556637 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":84,"threads_started":1}
I20260925 02:31:40.557334 15078 tablet_service.cc:1467] Processing CreateTablet for tablet e552cdefd02a4abe9875f98c7beddff6 (DEFAULT_TABLE table=test-table [id=b8146872ce4142968e0618aa626273ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:40.557363 15340 tablet_service.cc:1467] Processing CreateTablet for tablet e552cdefd02a4abe9875f98c7beddff6 (DEFAULT_TABLE table=test-table [id=b8146872ce4142968e0618aa626273ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:40.557731 15078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e552cdefd02a4abe9875f98c7beddff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:40.557731 15340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e552cdefd02a4abe9875f98c7beddff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:40.560678 15420 tablet_bootstrap.cc:492] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941: Bootstrap starting.
I20260925 02:31:40.561066 15209 tablet_service.cc:1467] Processing CreateTablet for tablet e552cdefd02a4abe9875f98c7beddff6 (DEFAULT_TABLE table=test-table [id=b8146872ce4142968e0618aa626273ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:40.561421 15209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e552cdefd02a4abe9875f98c7beddff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:40.561448 15420 tablet_bootstrap.cc:654] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:40.561748 15420 log.cc:824] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:40.562111 15421 tablet_bootstrap.cc:492] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: Bootstrap starting.
I20260925 02:31:40.562651 15421 tablet_bootstrap.cc:654] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:40.562927 15421 log.cc:824] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:40.563475 15420 tablet_bootstrap.cc:492] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941: No bootstrap required, opened a new log
I20260925 02:31:40.563561 15420 ts_tablet_manager.cc:1397] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:31:40.563692 15421 tablet_bootstrap.cc:492] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: No bootstrap required, opened a new log
I20260925 02:31:40.563766 15421 ts_tablet_manager.cc:1397] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:40.564466 15424 tablet_bootstrap.cc:492] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: Bootstrap starting.
I20260925 02:31:40.565133 15424 tablet_bootstrap.cc:654] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:40.565445 15424 log.cc:824] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:40.565999 15420 raft_consensus.cc:348] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.566166 15420 raft_consensus.cc:374] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:40.566208 15420 raft_consensus.cc:729] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4f30075edf4677b9126f561ed28941, State: Initialized, Role: FOLLOWER
I20260925 02:31:40.566313 15424 tablet_bootstrap.cc:492] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: No bootstrap required, opened a new log
I20260925 02:31:40.566380 15424 ts_tablet_manager.cc:1397] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:40.566340 15420 consensus_queue.cc:260] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.566644 15144 heartbeater.cc:503] Master 127.10.13.62:39345 was elected leader, sending a full tablet report...
I20260925 02:31:40.566574 15421 raft_consensus.cc:348] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.566716 15421 raft_consensus.cc:374] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:40.566758 15421 raft_consensus.cc:729] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959afa8883b2475b9f576ea7c85f8622, State: Initialized, Role: FOLLOWER
I20260925 02:31:40.566644 15420 ts_tablet_manager.cc:1428] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:40.566869 15421 consensus_queue.cc:260] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.567106 15421 ts_tablet_manager.cc:1428] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 02:31:40.567260 15406 heartbeater.cc:503] Master 127.10.13.62:39345 was elected leader, sending a full tablet report...
I20260925 02:31:40.568833 15424 raft_consensus.cc:348] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.569017 15424 raft_consensus.cc:374] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:40.569042 15424 raft_consensus.cc:729] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fe554c6fe64b3eb56822329bef8d36, State: Initialized, Role: FOLLOWER
I20260925 02:31:40.569144 15424 consensus_queue.cc:260] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.569361 15424 ts_tablet_manager.cc:1428] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:40.569440 15275 heartbeater.cc:503] Master 127.10.13.62:39345 was elected leader, sending a full tablet report...
I20260925 02:31:40.572623 15428 raft_consensus.cc:482] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:40.572731 15428 raft_consensus.cc:504] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.573102 15428 leader_election.cc:288] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 959afa8883b2475b9f576ea7c85f8622 (127.10.13.3:41413), 0e4f30075edf4677b9126f561ed28941 (127.10.13.1:36975)
I20260925 02:31:40.574400 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.573242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:41413 (local address 127.10.13.2:33143)
0925 02:31:40.573287 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.573323 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.573376 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.573521 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.573522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.573543 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:40.574039 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:31:40.574040 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:40.574047 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:31:40.574058 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:40.574253 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:40.574253 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:40.574254 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 02:31:40.574302 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:31:40.574316 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 02:31:40.574503 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.573336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:46423 (local address 127.10.13.1:36975)
0925 02:31:40.573402 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.573405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.573558 (+ 153us) server_negotiation.cc:407] Connection header received
0925 02:31:40.573595 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.573596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.573601 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.573660 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.574306 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.574307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.574307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.574312 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.574325 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.574329 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.574385 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.574418 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.574431 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:31:40.574579 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.573353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:33143 (local address 127.10.13.3:41413)
0925 02:31:40.573403 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.573405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.573414 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:40.573440 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.573440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.573446 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.573518 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.574154 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.574156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.574156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.574161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.574176 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.574180 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.574255 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.574492 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.574506 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:31:40.574534 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.573279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:36975 (local address 127.10.13.2:46423)
0925 02:31:40.573464 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.573488 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.573541 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.573662 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.573663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.573664 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:40.574221 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:31:40.574222 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:40.574229 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:31:40.574242 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:40.574384 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:40.574384 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:40.574385 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 02:31:40.574417 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 02:31:40.574428 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":109,"threads_started":1}
I20260925 02:31:40.574838 15360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e552cdefd02a4abe9875f98c7beddff6" candidate_uuid: "a2fe554c6fe64b3eb56822329bef8d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959afa8883b2475b9f576ea7c85f8622" is_pre_election: true
I20260925 02:31:40.574983 15360 raft_consensus.cc:2454] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2fe554c6fe64b3eb56822329bef8d36 in term 0.
I20260925 02:31:40.574957 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e552cdefd02a4abe9875f98c7beddff6" candidate_uuid: "a2fe554c6fe64b3eb56822329bef8d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4f30075edf4677b9126f561ed28941" is_pre_election: true
I20260925 02:31:40.575119 15098 raft_consensus.cc:2454] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2fe554c6fe64b3eb56822329bef8d36 in term 0.
I20260925 02:31:40.575234 15162 leader_election.cc:302] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 959afa8883b2475b9f576ea7c85f8622, a2fe554c6fe64b3eb56822329bef8d36; no voters:
I20260925 02:31:40.575366 15428 raft_consensus.cc:2781] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:40.575420 15428 raft_consensus.cc:482] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:40.575443 15428 raft_consensus.cc:3037] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:40.576146 15428 raft_consensus.cc:504] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.576289 15428 leader_election.cc:288] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [CANDIDATE]: Term 1 election: Requested vote from peers 959afa8883b2475b9f576ea7c85f8622 (127.10.13.3:41413), 0e4f30075edf4677b9126f561ed28941 (127.10.13.1:36975)
I20260925 02:31:40.576434 15360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e552cdefd02a4abe9875f98c7beddff6" candidate_uuid: "a2fe554c6fe64b3eb56822329bef8d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959afa8883b2475b9f576ea7c85f8622"
I20260925 02:31:40.576496 15360 raft_consensus.cc:3037] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:40.577430 15360 raft_consensus.cc:2454] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2fe554c6fe64b3eb56822329bef8d36 in term 1.
I20260925 02:31:40.577551 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e552cdefd02a4abe9875f98c7beddff6" candidate_uuid: "a2fe554c6fe64b3eb56822329bef8d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4f30075edf4677b9126f561ed28941"
I20260925 02:31:40.577591 15162 leader_election.cc:302] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 959afa8883b2475b9f576ea7c85f8622, a2fe554c6fe64b3eb56822329bef8d36; no voters:
I20260925 02:31:40.577633 15098 raft_consensus.cc:3037] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:40.577692 15428 raft_consensus.cc:2781] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:40.577855 15428 raft_consensus.cc:686] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 1 LEADER]: Becoming Leader. State: Replica: a2fe554c6fe64b3eb56822329bef8d36, State: Running, Role: LEADER
I20260925 02:31:40.577955 15428 consensus_queue.cc:237] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } }
I20260925 02:31:40.578567 15098 raft_consensus.cc:2454] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2fe554c6fe64b3eb56822329bef8d36 in term 1.
I20260925 02:31:40.578799 14955 catalog_manager.cc:5495] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2fe554c6fe64b3eb56822329bef8d36 (127.10.13.2). New cstate: current_term: 1 leader_uuid: "a2fe554c6fe64b3eb56822329bef8d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2fe554c6fe64b3eb56822329bef8d36" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 } health_report { overall_health: UNKNOWN } } }
I20260925 02:31:40.586251 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.585357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.10.13.2:44409)
0925 02:31:40.585402 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.585404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.585412 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:40.585488 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.585488 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.585499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.585544 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.586028 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.586029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.586029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.586033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.586047 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.586051 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.586094 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.586126 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.586142 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:40.587931 15098 raft_consensus.cc:1261] T e552cdefd02a4abe9875f98c7beddff6 P 0e4f30075edf4677b9126f561ed28941 [term 1 FOLLOWER]: Refusing update from remote peer a2fe554c6fe64b3eb56822329bef8d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:40.587941 15360 raft_consensus.cc:1261] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 1 FOLLOWER]: Refusing update from remote peer a2fe554c6fe64b3eb56822329bef8d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:40.588289 15428 consensus_queue.cc:1035] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "959afa8883b2475b9f576ea7c85f8622" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:40.588418 15430 consensus_queue.cc:1035] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4f30075edf4677b9126f561ed28941" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 36975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:40.591387 15434 mvcc.cc:205] Tried to move back new op lower bound from 7333083138405732352 to 7333083138367967232. Current Snapshot: MvccSnapshot[applied={T|T < 7333083138405732352}]
I20260925 02:31:40.593389 14955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:40.593488 14955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:40.594281 14955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20d08355be1d47e69233086fb83e4528: Sending DeleteTablet for 3 replicas of tablet e552cdefd02a4abe9875f98c7beddff6
I20260925 02:31:40.594642 15340 tablet_service.cc:1514] Processing DeleteTablet for tablet e552cdefd02a4abe9875f98c7beddff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20260925 02:31:40.594743 15209 tablet_service.cc:1514] Processing DeleteTablet for tablet e552cdefd02a4abe9875f98c7beddff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:40 UTC) from {username='slave'} at 127.0.0.1:56402
I20260925 02:31:40.594919 15078 tablet_service.cc:1514] Processing DeleteTablet for tablet e552cdefd02a4abe9875f98c7beddff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:40 UTC) from {username='slave'} at 127.0.0.1:35752
I20260925 02:31:40.594969 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15016
I20260925 02:31:40.595645 15443 tablet_replica.cc:330] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: stopping tablet replica
I20260925 02:31:40.595764 15443 raft_consensus.cc:2229] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:40.595875 15443 raft_consensus.cc:2258] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:40.596031 15444 tablet_replica.cc:330] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: stopping tablet replica
I20260925 02:31:40.596266 15444 raft_consensus.cc:2229] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:40.596398 15444 raft_consensus.cc:2258] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:40.596424 15443 ts_tablet_manager.cc:1905] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:40.597193 15444 ts_tablet_manager.cc:1905] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:40.598168 15443 ts_tablet_manager.cc:1918] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:40.598248 15443 log.cc:1196] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/e552cdefd02a4abe9875f98c7beddff6
I20260925 02:31:40.598518 15443 ts_tablet_manager.cc:1939] T e552cdefd02a4abe9875f98c7beddff6 P 959afa8883b2475b9f576ea7c85f8622: Deleting consensus metadata
I20260925 02:31:40.598965 14940 catalog_manager.cc:4841] TS 959afa8883b2475b9f576ea7c85f8622 (127.10.13.3:41413): tablet e552cdefd02a4abe9875f98c7beddff6 (table test-table [id=b8146872ce4142968e0618aa626273ef]) successfully deleted
I20260925 02:31:40.599215 15444 ts_tablet_manager.cc:1918] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:40.599301 15444 log.cc:1196] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/e552cdefd02a4abe9875f98c7beddff6
I20260925 02:31:40.599539 15444 ts_tablet_manager.cc:1939] T e552cdefd02a4abe9875f98c7beddff6 P a2fe554c6fe64b3eb56822329bef8d36: Deleting consensus metadata
I20260925 02:31:40.599954 14943 catalog_manager.cc:4841] TS a2fe554c6fe64b3eb56822329bef8d36 (127.10.13.2:44409): tablet e552cdefd02a4abe9875f98c7beddff6 (table test-table [id=b8146872ce4142968e0618aa626273ef]) successfully deleted
W20260925 02:31:40.603610 14941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:36975 (error 108)
I20260925 02:31:40.604171 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.604012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:36975 (local address 127.0.0.1:35758)
0925 02:31:40.604066 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.604109 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:31:40.604233 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15147
W20260925 02:31:40.608170 14941 catalog_manager.cc:4584] TS 0e4f30075edf4677b9126f561ed28941 (127.10.13.1:36975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e552cdefd02a4abe9875f98c7beddff6: Network error: Client connection negotiation failed: client connection to 127.10.13.1:36975: connect: Connection refused (error 111)
I20260925 02:31:40.611799 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15278
I20260925 02:31:40.616449 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 14925
2026-09-25T02:31:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (601 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 02:31:40.636273 10292 mini_oidc.cc:152] Starting JWKS server
I20260925 02:31:40.637537 10292 webserver.cc:460] Webserver started at https://127.0.0.1:44843/ using document root <none> and password file <none>
I20260925 02:31:40.637667 10292 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:31:40.637856 10292 webserver.cc:460] Webserver started at http://127.0.0.1:37443/ using document root <none> and password file <none>
2026-09-25T02:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:40Z Disabled control of system clock
I20260925 02:31:40.645866 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40159
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:40159
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44843/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/testchainca.cert with env {}
W20260925 02:31:40.726768 15452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:40.726967 15452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:40.726989 15452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:40.728152 15452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:40.728194 15452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:40.728206 15452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:40.728219 15452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:40.728231 15452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:31:40.728286 15452 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:31:40.728312 15452 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44843/jwks.json
I20260925 02:31:40.729954 15452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:40159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44843/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:40.730217 15452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:40.730445 15452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:40.735324 15457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.735317 15458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.735370 15460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:40.735705 15452 server_base.cc:1034] running on GCE node
I20260925 02:31:40.735980 15452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:40.736279 15452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:40.737486 15452 hybrid_clock.cc:648] HybridClock initialized: now 1790303500737477 us; error 40 us; skew 500 ppm
I20260925 02:31:40.738812 15452 webserver.cc:460] Webserver started at http://127.10.13.62:34149/ using document root <none> and password file <none>
I20260925 02:31:40.739084 15452 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:40.739163 15452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:40.739327 15452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:40.740419 15452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "5bd2d8e951584feaabfb15a470a15e06"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e276d74424e2092218257cf9d525fd51"
server_key_iv: "1f95ec126e74a3e49052bd3a20873091"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.740782 15452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "5bd2d8e951584feaabfb15a470a15e06"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e276d74424e2092218257cf9d525fd51"
server_key_iv: "1f95ec126e74a3e49052bd3a20873091"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.742117 15452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:40.742905 15466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.743088 15452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:40.743196 15452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "5bd2d8e951584feaabfb15a470a15e06"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e276d74424e2092218257cf9d525fd51"
server_key_iv: "1f95ec126e74a3e49052bd3a20873091"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.743273 15452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:40.761264 15452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:40.761600 15452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:40.761761 15452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:40.765879 15452 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:40159
I20260925 02:31:40.765950 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:40159 every 8 connection(s)
I20260925 02:31:40.766268 15452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:40.767288 15519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:40.769701 15519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06: Bootstrap starting.
I20260925 02:31:40.770247 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15452
I20260925 02:31:40.770319 15519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:40.770370 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:40.770553 10292 external_mini_cluster.cc:1428] Setting key c85cfd6e0ec82308320f56d3ff0fd77b
I20260925 02:31:40.770665 15519 log.cc:824] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:40.771482 15519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06: No bootstrap required, opened a new log
I20260925 02:31:40.773526 15519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd2d8e951584feaabfb15a470a15e06" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40159 } }
I20260925 02:31:40.773687 15519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:40.773734 15519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd2d8e951584feaabfb15a470a15e06, State: Initialized, Role: FOLLOWER
I20260925 02:31:40.773907 15519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd2d8e951584feaabfb15a470a15e06" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40159 } }
I20260925 02:31:40.774017 15519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:40.774061 15519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:40.774111 15519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:40.774672 15519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd2d8e951584feaabfb15a470a15e06" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40159 } }
I20260925 02:31:40.774786 15519 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd2d8e951584feaabfb15a470a15e06; no voters:
I20260925 02:31:40.774957 15519 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:40.775032 15524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:40.775198 15524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd2d8e951584feaabfb15a470a15e06, State: Running, Role: LEADER
I20260925 02:31:40.775238 15519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:40.775300 15524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd2d8e951584feaabfb15a470a15e06" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40159 } }
I20260925 02:31:40.775600 15524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd2d8e951584feaabfb15a470a15e06. Latest consensus state: current_term: 1 leader_uuid: "5bd2d8e951584feaabfb15a470a15e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd2d8e951584feaabfb15a470a15e06" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40159 } } }
I20260925 02:31:40.775614 15525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd2d8e951584feaabfb15a470a15e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd2d8e951584feaabfb15a470a15e06" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40159 } } }
I20260925 02:31:40.775688 15524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:40.775692 15525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:40.775938 15530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:40.776460 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.771107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.10.13.62:40159)
0925 02:31:40.771319 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.771328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.771364 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:31:40.771858 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.771865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.771927 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.772077 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.772301 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:40.773220 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:40.774172 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:40.774581 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:40.774673 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:40.775576 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.775580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.775581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.775808 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.775856 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.775874 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.775921 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.776193 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.776282 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260925 02:31:40.776468 15530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:40.777452 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:31:40.778829 15530 catalog_manager.cc:1347] Generated new cluster ID: 56fa271909ef4e21b4f878ca1f127da5
I20260925 02:31:40.778878 15530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:40.789040 15530 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:40.789765 15530 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:40.801069 15530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd2d8e951584feaabfb15a470a15e06: Generated new TSK 0
I20260925 02:31:40.801282 15530 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:31:40.827688 15549 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 02:31:40.827802 15549 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:31:40.828030 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.812906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.10.13.62:40159)
0925 02:31:40.812973 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.812976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.813125 (+ 149us) server_negotiation.cc:407] Connection header received
0925 02:31:40.813230 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.813230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.813241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.813299 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:31:40.813442 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:40.813942 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:40.815021 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:40.815214 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:40.815262 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:40.815495 (+ 233us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:31:40.827783 (+ 12288us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:31:40.827950 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56544: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 02:31:40.828110 15523 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56544: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 02:31:40.828414 15548 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:40.812817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40159 (local address 127.0.0.1:56544)
0925 02:31:40.813047 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:40.813059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:40.813134 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:40.813305 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:40.813306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:40.813307 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:31:40.813381 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:40.813383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:40.814024 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:40.814025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:40.814927 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:40.814930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:40.815299 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:40.815301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:40.815368 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:40.815390 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:31:40.827994 (+ 12604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:31:40.828114 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.13.62:40159: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.13.62:40159: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 02:31:40.828836 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15452
2026-09-25T02:31:40Z chronyd exiting
I20260925 02:31:40.865145 10292 test_util.cc:175] -----------------------------------------------
I20260925 02:31:40.865223 10292 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790303478591324-10292-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 02:31:40.865928 10292 mini_oidc.cc:152] Starting JWKS server
I20260925 02:31:40.866564 10292 webserver.cc:460] Webserver started at https://127.0.0.1:38289/ using document root <none> and password file <none>
I20260925 02:31:40.866601 10292 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:31:40.866719 10292 webserver.cc:460] Webserver started at http://127.0.0.1:40807/ using document root <none> and password file <none>
2026-09-25T02:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:40Z Disabled control of system clock
I20260925 02:31:40.874536 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39635
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39635
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38289/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/testchainca.cert with env {}
W20260925 02:31:40.953872 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:40.954085 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:40.954107 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:40.955220 15556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:40.955256 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:40.955269 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:40.955281 15556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:40.955293 15556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:31:40.955305 15556 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:31:40.955317 15556 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38289/jwks.json
I20260925 02:31:40.956756 15556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38289/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:40.957028 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:40.957199 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:40.961949 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.961974 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:40.962006 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:40.962208 15556 server_base.cc:1034] running on GCE node
I20260925 02:31:40.962436 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:40.962682 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:40.963824 15556 hybrid_clock.cc:648] HybridClock initialized: now 1790303500963801 us; error 35 us; skew 500 ppm
I20260925 02:31:40.965080 15556 webserver.cc:460] Webserver started at http://127.10.13.62:35703/ using document root <none> and password file <none>
I20260925 02:31:40.965296 15556 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:40.965335 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:40.965420 15556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:40.966514 15556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "76b30116d7254fa9abd7ae31ef1904d4"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e9a40565a7085c1082c3d5772a0817ff"
server_key_iv: "8398e90ee80974a3d901624977d74906"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.966827 15556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "76b30116d7254fa9abd7ae31ef1904d4"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e9a40565a7085c1082c3d5772a0817ff"
server_key_iv: "8398e90ee80974a3d901624977d74906"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.968110 15556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:40.968773 15570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:40.968940 15556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:40.969017 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "76b30116d7254fa9abd7ae31ef1904d4"
format_stamp: "Formatted at 2026-09-25 02:31:40 on dist-test-slave-q8mg"
server_key: "e9a40565a7085c1082c3d5772a0817ff"
server_key_iv: "8398e90ee80974a3d901624977d74906"
server_key_version: "encryptionkey@0"
I20260925 02:31:40.969072 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:40.982575 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:40.982810 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:40.982910 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:40.986656 15556 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:39635
I20260925 02:31:40.986689 15622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:39635 every 8 connection(s)
I20260925 02:31:40.987046 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:40.987792 15623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:40.988433 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15556
I20260925 02:31:40.988559 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:40.988788 10292 external_mini_cluster.cc:1428] Setting key c38e2f4f8d22763aa8e9ff5d00223dd5
I20260925 02:31:40.991703 15623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4: Bootstrap starting.
I20260925 02:31:40.992334 15623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:40.992645 15623 log.cc:824] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:40.993384 15623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4: No bootstrap required, opened a new log
I20260925 02:31:40.994364 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:40.989345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.10.13.62:39635)
0925 02:31:40.989571 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:40.989578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:40.989602 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:31:40.990021 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:40.990028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:40.990086 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:40.990240 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:40.990500 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:40.991541 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:40.992410 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:40.992862 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:40.993046 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:40.993719 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:40.993724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:40.993724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:40.993894 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:40.993939 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:40.993949 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:40.994029 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:40.994105 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 02:31:40.994186 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260925 02:31:40.995407 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:31:40.995641 15623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b30116d7254fa9abd7ae31ef1904d4" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39635 } }
I20260925 02:31:40.995796 15623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:40.995842 15623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b30116d7254fa9abd7ae31ef1904d4, State: Initialized, Role: FOLLOWER
I20260925 02:31:40.996078 15623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b30116d7254fa9abd7ae31ef1904d4" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39635 } }
I20260925 02:31:40.996181 15623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:40.996227 15623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:40.996277 15623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:40.997104 15623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b30116d7254fa9abd7ae31ef1904d4" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39635 } }
I20260925 02:31:40.997247 15623 leader_election.cc:302] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b30116d7254fa9abd7ae31ef1904d4; no voters:
I20260925 02:31:40.997422 15623 leader_election.cc:288] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:40.997478 15628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:40.997648 15628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [term 1 LEADER]: Becoming Leader. State: Replica: 76b30116d7254fa9abd7ae31ef1904d4, State: Running, Role: LEADER
I20260925 02:31:40.997686 15623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:40.997740 15628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b30116d7254fa9abd7ae31ef1904d4" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39635 } }
I20260925 02:31:40.998095 15628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b30116d7254fa9abd7ae31ef1904d4. Latest consensus state: current_term: 1 leader_uuid: "76b30116d7254fa9abd7ae31ef1904d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b30116d7254fa9abd7ae31ef1904d4" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39635 } } }
I20260925 02:31:40.998185 15628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:40.998106 15629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76b30116d7254fa9abd7ae31ef1904d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b30116d7254fa9abd7ae31ef1904d4" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39635 } } }
I20260925 02:31:40.998315 15629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:40.998989 15635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:40.999476 15635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:41.001020 15635 catalog_manager.cc:1347] Generated new cluster ID: 0c36daa73d4845d7972a6e8a961ad20b
I20260925 02:31:41.001081 15635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:41.008718 15635 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:41.009407 15635 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:41.016214 15635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76b30116d7254fa9abd7ae31ef1904d4: Generated new TSK 0
I20260925 02:31:41.016389 15635 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:31:41.038355 15654 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 02:31:41.038448 15654 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:31:41.038792 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:41.024496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.10.13.62:39635)
0925 02:31:41.024542 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:41.024544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:41.024651 (+ 107us) server_negotiation.cc:407] Connection header received
0925 02:31:41.024689 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:41.024690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:41.024704 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:41.024784 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:31:41.024929 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.025725 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.026489 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.026809 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.026877 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:41.027024 (+ 147us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:31:41.038558 (+ 11534us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:31:41.038720 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43854: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 02:31:41.038851 15626 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43854: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 02:31:41.038863 15653 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:41.024423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39635 (local address 127.0.0.1:43854)
0925 02:31:41.024567 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:41.024578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:41.024640 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:41.024788 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:41.024788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:41.024790 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:31:41.024839 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:41.024841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.025830 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.025830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:41.026407 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:41.026409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.026884 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.026885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:41.026924 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:41.026929 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:31:41.038668 (+ 11739us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:31:41.038770 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.13.62:39635: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260925 02:31:41.039171 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15556
2026-09-25T02:31:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 02:31:41.077176 10292 mini_oidc.cc:152] Starting JWKS server
I20260925 02:31:41.077852 10292 webserver.cc:460] Webserver started at https://127.0.0.1:44249/ using document root <none> and password file <none>
I20260925 02:31:41.077893 10292 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:31:41.078006 10292 webserver.cc:460] Webserver started at http://127.0.0.1:41245/ using document root <none> and password file <none>
2026-09-25T02:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:41Z Disabled control of system clock
I20260925 02:31:41.086283 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:35613
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:35613
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44249/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:31:41.168424 15660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:41.168650 15660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:41.168673 15660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:41.169884 15660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:41.169940 15660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:41.169965 15660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:41.169978 15660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:41.169991 15660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:31:41.170003 15660 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:31:41.170015 15660 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44249/jwks.json
I20260925 02:31:41.171394 15660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:35613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44249/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:41.171610 15660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:41.171877 15660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:41.177125 15666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:41.177125 15668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:41.177119 15665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:41.177388 15660 server_base.cc:1034] running on GCE node
I20260925 02:31:41.177671 15660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:41.177976 15660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:41.179177 15660 hybrid_clock.cc:648] HybridClock initialized: now 1790303501179159 us; error 26 us; skew 500 ppm
I20260925 02:31:41.180541 15660 webserver.cc:460] Webserver started at http://127.10.13.62:45463/ using document root <none> and password file <none>
I20260925 02:31:41.180795 15660 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:41.180866 15660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:41.181061 15660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:41.182201 15660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "8de584ce189d4799a7560ac28d26a35b"
format_stamp: "Formatted at 2026-09-25 02:31:41 on dist-test-slave-q8mg"
server_key: "f6f2e1b86640edb8e814c58d3b1c2f29"
server_key_iv: "956a1df1f51ed71198b27e3a4a217900"
server_key_version: "encryptionkey@0"
I20260925 02:31:41.182579 15660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "8de584ce189d4799a7560ac28d26a35b"
format_stamp: "Formatted at 2026-09-25 02:31:41 on dist-test-slave-q8mg"
server_key: "f6f2e1b86640edb8e814c58d3b1c2f29"
server_key_iv: "956a1df1f51ed71198b27e3a4a217900"
server_key_version: "encryptionkey@0"
I20260925 02:31:41.183911 15660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:41.184635 15674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:41.184813 15660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:41.184942 15660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "8de584ce189d4799a7560ac28d26a35b"
format_stamp: "Formatted at 2026-09-25 02:31:41 on dist-test-slave-q8mg"
server_key: "f6f2e1b86640edb8e814c58d3b1c2f29"
server_key_iv: "956a1df1f51ed71198b27e3a4a217900"
server_key_version: "encryptionkey@0"
I20260925 02:31:41.185045 15660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:41.206950 15660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:41.207269 15660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:41.207427 15660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:41.211704 15660 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:35613
I20260925 02:31:41.211753 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:35613 every 8 connection(s)
I20260925 02:31:41.212136 15660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:41.212843 15727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:41.215715 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b: Bootstrap starting.
I20260925 02:31:41.216264 15727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:41.216508 15727 log.cc:824] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:41.217224 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b: No bootstrap required, opened a new log
I20260925 02:31:41.218791 15727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de584ce189d4799a7560ac28d26a35b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35613 } }
I20260925 02:31:41.218925 15727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:41.218948 15727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de584ce189d4799a7560ac28d26a35b, State: Initialized, Role: FOLLOWER
I20260925 02:31:41.219053 15727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de584ce189d4799a7560ac28d26a35b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35613 } }
I20260925 02:31:41.219134 15727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:41.219168 15727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:41.219219 15727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:41.219759 15727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de584ce189d4799a7560ac28d26a35b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35613 } }
I20260925 02:31:41.219884 15727 leader_election.cc:302] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8de584ce189d4799a7560ac28d26a35b; no voters:
I20260925 02:31:41.220064 15727 leader_election.cc:288] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:41.220178 15730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:41.220346 15727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:41.220367 15730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [term 1 LEADER]: Becoming Leader. State: Replica: 8de584ce189d4799a7560ac28d26a35b, State: Running, Role: LEADER
I20260925 02:31:41.220516 15730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de584ce189d4799a7560ac28d26a35b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35613 } }
I20260925 02:31:41.220785 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15660
I20260925 02:31:41.220940 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:41.221007 15732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8de584ce189d4799a7560ac28d26a35b. Latest consensus state: current_term: 1 leader_uuid: "8de584ce189d4799a7560ac28d26a35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de584ce189d4799a7560ac28d26a35b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35613 } } }
I20260925 02:31:41.221084 15732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:41.221176 10292 external_mini_cluster.cc:1428] Setting key dcd8cb924c6ac792c23eefa711360503
I20260925 02:31:41.221477 15731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8de584ce189d4799a7560ac28d26a35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de584ce189d4799a7560ac28d26a35b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35613 } } }
I20260925 02:31:41.221552 15731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:41.221848 15739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:41.222412 15739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:41.224103 15739 catalog_manager.cc:1347] Generated new cluster ID: 371ffcc5c8a148778858b07e202768cd
I20260925 02:31:41.224174 15739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:41.227399 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:41.221887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.10.13.62:35613)
0925 02:31:41.222116 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:41.222125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:41.222158 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:31:41.222277 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:41.222284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:41.222359 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:41.222520 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:41.222758 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.223952 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.225054 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.225754 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.225893 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:41.226626 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:41.226630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:41.226631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:41.226858 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:41.226908 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:41.226924 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:41.227003 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:41.227068 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 02:31:41.227156 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 02:31:41.228438 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:31:41.232980 15739 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:41.233520 15739 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:41.238355 15739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8de584ce189d4799a7560ac28d26a35b: Generated new TSK 0
I20260925 02:31:41.238559 15739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:31:41.248728 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:41.244833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.10.13.62:35613)
0925 02:31:41.244908 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:41.244910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:41.245017 (+ 107us) server_negotiation.cc:407] Connection header received
0925 02:31:41.245089 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:41.245090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:41.245102 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:41.245171 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:31:41.245353 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.245902 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.247200 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.247402 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.247459 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:41.247678 (+ 219us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:31:41.248485 (+ 807us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44249/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:31:41.248649 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60162: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44249/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 02:31:41.248775 15756 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:41.244714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:35613 (local address 127.0.0.1:60162)
0925 02:31:41.244929 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:41.244942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:41.245014 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:41.245190 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:41.245191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:41.245192 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:31:41.245263 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:41.245265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.246017 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.246018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:41.247095 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:41.247098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.247491 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.247492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:41.247560 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:41.247566 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:31:41.248590 (+ 1024us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44249/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:31:41.248681 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.13.62:35613: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44249/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":116,"threads_started":1}
W20260925 02:31:41.248791 15748 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60162: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44249/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260925 02:31:41.249231 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15660
E20260925 02:31:41.262830 15757 webserver.cc:533] Webserver: sslize failed:
E20260925 02:31:41.262912 15757 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T02:31:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T02:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:41Z Disabled control of system clock
I20260925 02:31:41.296345 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45955
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:45955
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/testchainca.cert with env {}
W20260925 02:31:41.377703 15761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:41.377916 15761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:41.377940 15761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:41.379142 15761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:41.379186 15761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:41.379199 15761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:41.379211 15761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:41.379262 15761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:31:41.379280 15761 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:31:41.379304 15761 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 02:31:41.380998 15761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:45955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:41.381274 15761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:41.381503 15761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:41.386397 15767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:41.386397 15769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:41.386438 15766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:41.386802 15761 server_base.cc:1034] running on GCE node
I20260925 02:31:41.387017 15761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:41.387293 15761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:41.388437 15761 hybrid_clock.cc:648] HybridClock initialized: now 1790303501388433 us; error 44 us; skew 500 ppm
I20260925 02:31:41.389726 15761 webserver.cc:460] Webserver started at http://127.10.13.62:42925/ using document root <none> and password file <none>
I20260925 02:31:41.389936 15761 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:41.389977 15761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:41.390070 15761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:41.391170 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "645a106cc42146709c0d83489d89d3fe"
format_stamp: "Formatted at 2026-09-25 02:31:41 on dist-test-slave-q8mg"
server_key: "dfc4d91063730280477f4d18a0855dca"
server_key_iv: "ef6f27084f70b6aff88112aed6e143eb"
server_key_version: "encryptionkey@0"
I20260925 02:31:41.391471 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "645a106cc42146709c0d83489d89d3fe"
format_stamp: "Formatted at 2026-09-25 02:31:41 on dist-test-slave-q8mg"
server_key: "dfc4d91063730280477f4d18a0855dca"
server_key_iv: "ef6f27084f70b6aff88112aed6e143eb"
server_key_version: "encryptionkey@0"
I20260925 02:31:41.392736 15761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:41.393555 15775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:41.393749 15761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:41.393832 15761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "645a106cc42146709c0d83489d89d3fe"
format_stamp: "Formatted at 2026-09-25 02:31:41 on dist-test-slave-q8mg"
server_key: "dfc4d91063730280477f4d18a0855dca"
server_key_iv: "ef6f27084f70b6aff88112aed6e143eb"
server_key_version: "encryptionkey@0"
I20260925 02:31:41.393887 15761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:41.421653 15761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:41.421958 15761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:41.422067 15761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:41.425927 15761 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:45955
I20260925 02:31:41.425997 15828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:45955 every 8 connection(s)
I20260925 02:31:41.426347 15761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:41.427372 15829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:41.429791 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe: Bootstrap starting.
I20260925 02:31:41.430388 15829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:41.430689 15829 log.cc:824] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:41.430912 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15761
I20260925 02:31:41.431023 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:41.431229 10292 external_mini_cluster.cc:1428] Setting key f5eef33a495928aa6d5567328aaf77e0
I20260925 02:31:41.431576 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe: No bootstrap required, opened a new log
I20260925 02:31:41.434481 15829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645a106cc42146709c0d83489d89d3fe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45955 } }
I20260925 02:31:41.434664 15829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:41.434695 15829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645a106cc42146709c0d83489d89d3fe, State: Initialized, Role: FOLLOWER
I20260925 02:31:41.434816 15829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645a106cc42146709c0d83489d89d3fe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45955 } }
I20260925 02:31:41.434906 15829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:41.434947 15829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:41.435002 15829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:41.435595 15829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645a106cc42146709c0d83489d89d3fe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45955 } }
I20260925 02:31:41.435734 15829 leader_election.cc:302] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 645a106cc42146709c0d83489d89d3fe; no voters:
I20260925 02:31:41.435945 15829 leader_election.cc:288] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:41.435971 15834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:41.436093 15834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [term 1 LEADER]: Becoming Leader. State: Replica: 645a106cc42146709c0d83489d89d3fe, State: Running, Role: LEADER
I20260925 02:31:41.436184 15834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645a106cc42146709c0d83489d89d3fe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45955 } }
I20260925 02:31:41.436242 15829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:41.436523 15834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 645a106cc42146709c0d83489d89d3fe. Latest consensus state: current_term: 1 leader_uuid: "645a106cc42146709c0d83489d89d3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645a106cc42146709c0d83489d89d3fe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45955 } } }
I20260925 02:31:41.436604 15834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:41.436532 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "645a106cc42146709c0d83489d89d3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645a106cc42146709c0d83489d89d3fe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45955 } } }
I20260925 02:31:41.436764 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:41.437112 15839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:41.437669 15839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:41.438766 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:41.431809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.10.13.62:45955)
0925 02:31:41.432091 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:41.432100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:41.432144 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:31:41.432937 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:41.432948 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:41.433030 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:41.433218 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:41.433437 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.434847 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.435739 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.436380 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.436495 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:41.438089 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:41.438093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:41.438093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:31:41.438282 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:41.438314 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:31:41.438326 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:41.438402 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:41.438481 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 02:31:41.438565 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260925 02:31:41.439574 15839 catalog_manager.cc:1347] Generated new cluster ID: f0e5110de5c348c981661b3032959d60
I20260925 02:31:41.439659 15839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:41.439723 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:31:41.451009 15839 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:41.451769 15839 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:41.457348 15839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 645a106cc42146709c0d83489d89d3fe: Generated new TSK 0
I20260925 02:31:41.457587 15839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:31:41.919940 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:41.463290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.10.13.62:45955)
0925 02:31:41.463359 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:41.463361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:41.463476 (+ 115us) server_negotiation.cc:407] Connection header received
0925 02:31:41.463558 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:41.463559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:41.463572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:41.463659 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:31:41.463834 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.464329 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.465424 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.465645 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.465721 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:41.465906 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:31:41.919640 (+453734us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:31:41.919845 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46592: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 02:31:41.920032 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:41.463210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45955 (local address 127.0.0.1:46592)
0925 02:31:41.463404 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:41.463414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:41.463494 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:41.463673 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:41.463674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:41.463676 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:31:41.463756 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:41.463758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.464666 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.464667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:41.465339 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:41.465341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:41.465728 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:41.465729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:41.465792 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:41.465804 (+ 12us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:31:41.919798 (+453994us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:31:41.919920 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.13.62:45955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":114,"threads_started":1}
W20260925 02:31:41.920037 15833 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46592: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260925 02:31:41.920605 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15761
2026-09-25T02:31:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (650 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:42 dist-test-slave-q8mg krb5kdc[15870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:42 dist-test-slave-q8mg krb5kdc[15870](info): set up 2 sockets
Sep 25 02:31:42 dist-test-slave-q8mg krb5kdc[15870](info): commencing operation
krb5kdc: starting...
W20260925 02:31:43.151696 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:43.271716 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:43773
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:43773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:43.356401 15883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:43.356609 15883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:43.356631 15883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:43.357808 15883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:43.357851 15883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:43.357864 15883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:43.357904 15883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:43.357934 15883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:43.359365 15883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:43773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:43773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:43.359620 15883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:43.359845 15883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:31:43.365021 15883 server_base.cc:1034] running on GCE node
W20260925 02:31:43.364961 15888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:43.364985 15889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:43.364955 15891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:43.365477 15883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:43.365828 15883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:43.367000 15883 hybrid_clock.cc:648] HybridClock initialized: now 1790303503366992 us; error 47 us; skew 500 ppm
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.369490 15883 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:43.369946 15883 webserver.cc:460] Webserver started at http://127.10.13.62:38517/ using document root <none> and password file <none>
I20260925 02:31:43.370180 15883 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:43.370249 15883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:43.370407 15883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:43.371515 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "29ff8bc7500747eca6f92eb8a0de289e"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "ce76c1300f3499d88f66390c71d2a704"
server_key_iv: "bc522edc8ddc37a50cb7ba592bc3510c"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.371888 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "29ff8bc7500747eca6f92eb8a0de289e"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "ce76c1300f3499d88f66390c71d2a704"
server_key_iv: "bc522edc8ddc37a50cb7ba592bc3510c"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.373368 15883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:31:43.374166 15898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.374358 15883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:43.374464 15883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "29ff8bc7500747eca6f92eb8a0de289e"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "ce76c1300f3499d88f66390c71d2a704"
server_key_iv: "bc522edc8ddc37a50cb7ba592bc3510c"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.374552 15883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:43.395505 15883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:43.397722 15883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:43.397913 15883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:43.401973 15883 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:43773
I20260925 02:31:43.402005 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:43773 every 8 connection(s)
I20260925 02:31:43.402405 15883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:43.403337 15952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:43.405856 15952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e: Bootstrap starting.
I20260925 02:31:43.406373 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15883
I20260925 02:31:43.406464 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:43.406527 15952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:43.406680 10292 external_mini_cluster.cc:1428] Setting key e45ceb1a251eb3f2a54c13265bf88d2e
I20260925 02:31:43.406879 15952 log.cc:824] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:43.407693 15952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e: No bootstrap required, opened a new log
I20260925 02:31:43.410121 15952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8bc7500747eca6f92eb8a0de289e" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43773 } }
I20260925 02:31:43.410318 15952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:43.410355 15952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ff8bc7500747eca6f92eb8a0de289e, State: Initialized, Role: FOLLOWER
I20260925 02:31:43.410490 15952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8bc7500747eca6f92eb8a0de289e" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43773 } }
I20260925 02:31:43.410565 15952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:43.410594 15952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:43.410637 15952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:43.411281 15952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8bc7500747eca6f92eb8a0de289e" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43773 } }
I20260925 02:31:43.411401 15952 leader_election.cc:302] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ff8bc7500747eca6f92eb8a0de289e; no voters:
I20260925 02:31:43.411593 15952 leader_election.cc:288] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:43.411674 15957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:43.411849 15957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [term 1 LEADER]: Becoming Leader. State: Replica: 29ff8bc7500747eca6f92eb8a0de289e, State: Running, Role: LEADER
I20260925 02:31:43.412005 15957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8bc7500747eca6f92eb8a0de289e" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43773 } }
I20260925 02:31:43.412285 15952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:43.412388 15958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29ff8bc7500747eca6f92eb8a0de289e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8bc7500747eca6f92eb8a0de289e" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43773 } } }
I20260925 02:31:43.412467 15958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:43.412693 15959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ff8bc7500747eca6f92eb8a0de289e. Latest consensus state: current_term: 1 leader_uuid: "29ff8bc7500747eca6f92eb8a0de289e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8bc7500747eca6f92eb8a0de289e" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43773 } } }
I20260925 02:31:43.412810 15959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:43.413162 15961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:43.413697 15961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:43.415468 15961 catalog_manager.cc:1347] Generated new cluster ID: 2e20f288b2fd489d83e540dc1ab491f0
I20260925 02:31:43.415547 15961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:43.418646 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.407210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.10.13.62:43773)
0925 02:31:43.407397 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.407407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.407446 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:31:43.407924 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.407931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.407979 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.408096 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.408967 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.409843 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.410906 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.411313 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.411448 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.414767 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:43.414771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:43.414772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:43.414785 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:43.416987 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.417402 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.417406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.417406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.417516 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.417715 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.417716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.417716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.418043 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:43.418218 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:43.418289 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 02:31:43.418425 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260925 02:31:43.422691 15961 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:43.423334 15961 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:43.428011 15961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29ff8bc7500747eca6f92eb8a0de289e: Generated new TSK 0
I20260925 02:31:43.428196 15961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:43.457016 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43773
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:43.541883 15978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:43.542098 15978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:43.542124 15978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:43.543243 15978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:43.543282 15978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:43.543308 15978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:43.544737 15978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:43.544993 15978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:43.545204 15978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:43.550380 15986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:43.550380 15983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:43.550433 15984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:43.550927 15978 server_base.cc:1034] running on GCE node
I20260925 02:31:43.551225 15978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:43.551520 15978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:43.552718 15978 hybrid_clock.cc:648] HybridClock initialized: now 1790303503552703 us; error 31 us; skew 500 ppm
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.556094 15978 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:43.556708 15978 webserver.cc:460] Webserver started at http://127.10.13.1:44943/ using document root <none> and password file <none>
I20260925 02:31:43.557013 15978 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:43.557081 15978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:43.557233 15978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:43.558518 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "aebd796237f64d29a9ce61af897ee10b"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "aaf74f96c2de11f8a9faf868c901663c"
server_key_iv: "ea39c6657e0b48175296d126cec27b67"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.558825 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "aebd796237f64d29a9ce61af897ee10b"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "aaf74f96c2de11f8a9faf868c901663c"
server_key_iv: "ea39c6657e0b48175296d126cec27b67"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.560153 15978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:43.560966 15993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.561174 15978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:43.561273 15978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "aebd796237f64d29a9ce61af897ee10b"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "aaf74f96c2de11f8a9faf868c901663c"
server_key_iv: "ea39c6657e0b48175296d126cec27b67"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.561352 15978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:43.574437 15978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:43.576265 15978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:43.576375 15978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:43.576571 15978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:43.576846 15978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:43.576967 15978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.576995 15978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:43.577008 15978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.583279 15978 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:41177
I20260925 02:31:43.583323 16109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:41177 every 8 connection(s)
I20260925 02:31:43.583652 15978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:43.592576 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 15978
I20260925 02:31:43.592684 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:43.592914 10292 external_mini_cluster.cc:1428] Setting key 80dd65bce8f43bd283d0d242e32b4c16
I20260925 02:31:43.593186 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.584415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:40285 (local address 127.10.13.62:43773)
0925 02:31:43.584485 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.584487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.584726 (+ 239us) server_negotiation.cc:407] Connection header received
0925 02:31:43.585727 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.585728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.585739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.585795 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.586369 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.586896 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.587978 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.588205 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.588263 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.590330 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:43.590331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:43.590331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:43.590339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:43.591721 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.592082 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.592083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.592084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.592140 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.592356 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.592356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.592356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.592474 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:43.592530 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:43.592952 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 02:31:43.593051 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:31:43.593293 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.584405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43773 (local address 127.10.13.1:40285)
0925 02:31:43.584590 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:43.584641 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:43.585612 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:43.585849 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:43.585852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:43.586096 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:43.586288 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:43.586292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.586975 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.586976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:43.587902 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:43.587905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.588286 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.588286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:43.588382 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.588826 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:43.588835 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:43.590239 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:43.591795 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:43.591802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:43.591803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:43.592010 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:43.592203 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:43.592203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:43.592204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:43.592298 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:43.592545 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:43.592552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:43.592742 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 02:31:43.592886 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 02:31:43.593064 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260925 02:31:43.594201 16110 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43773
I20260925 02:31:43.594329 16110 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:43.594522 16110 heartbeater.cc:511] Master 127.10.13.62:43773 requested a full tablet report, sending...
I20260925 02:31:43.595216 15916 ts_manager.cc:194] Registered new tserver with Master: aebd796237f64d29a9ce61af897ee10b (127.10.13.1:41177)
I20260925 02:31:43.596163 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:40285
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:43.619237 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43773
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:43.702069 16115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:43.702273 16115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:43.702296 16115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:43.703512 16115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:43.703573 16115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:43.703599 16115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:43.705163 16115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:43.705439 16115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:43.705686 16115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:43.710777 16120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:43.710775 16123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:43.711024 16115 server_base.cc:1034] running on GCE node
W20260925 02:31:43.711082 16121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:43.711339 16115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:43.711568 16115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:43.712719 16115 hybrid_clock.cc:648] HybridClock initialized: now 1790303503712696 us; error 35 us; skew 500 ppm
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.715710 16115 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:43.716203 16115 webserver.cc:460] Webserver started at http://127.10.13.2:34345/ using document root <none> and password file <none>
I20260925 02:31:43.716387 16115 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:43.716427 16115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:43.716514 16115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:43.717574 16115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "1a5df011198d40e2a8d5b455c64625ee"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "4a21901c4257b9faf33b715a81d4d52f"
server_key_iv: "60e94f5528141333af2844d730d7be7c"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.717869 16115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "1a5df011198d40e2a8d5b455c64625ee"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "4a21901c4257b9faf33b715a81d4d52f"
server_key_iv: "60e94f5528141333af2844d730d7be7c"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.719216 16115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:43.719900 16130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.720113 16115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:43.720202 16115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "1a5df011198d40e2a8d5b455c64625ee"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "4a21901c4257b9faf33b715a81d4d52f"
server_key_iv: "60e94f5528141333af2844d730d7be7c"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.720257 16115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:43.742697 16115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:43.744930 16115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:43.745095 16115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:43.745301 16115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:43.745558 16115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:43.745589 16115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.745609 16115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:43.745622 16115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.751369 16115 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:40557
I20260925 02:31:43.751436 16244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:40557 every 8 connection(s)
I20260925 02:31:43.751690 16115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:43.754535 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 16115
I20260925 02:31:43.754660 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:43.754870 10292 external_mini_cluster.cc:1428] Setting key 600bba36687d93d0d9115b70abfeff05
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:43.762454 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.752438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43773 (local address 127.10.13.2:50027)
0925 02:31:43.752589 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:43.752623 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:43.753597 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:43.753838 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:43.753841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:43.754130 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:43.754366 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:43.754373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.755040 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.755042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:43.756241 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:43.756244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.756614 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.756616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:43.756764 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.757405 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:43.757414 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:43.758817 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:43.760815 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:43.760822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:43.760824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:43.761182 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:43.761442 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:43.761443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:43.761444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:43.761608 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:43.761940 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:43.761946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:43.762072 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 02:31:43.762151 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 02:31:43.762278 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260925 02:31:43.762451 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.752453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:50027 (local address 127.10.13.62:43773)
0925 02:31:43.752502 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.752505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.752689 (+ 184us) server_negotiation.cc:407] Connection header received
0925 02:31:43.753706 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.753707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.753718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.753779 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.754455 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.754942 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.756331 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.756522 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.756582 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.758915 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:43.758917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:43.758917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:43.758926 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:43.760721 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.761267 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.761268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.761269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.761336 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.761708 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.761709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.761709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.761865 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:43.761952 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:43.762228 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 02:31:43.762319 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:43.763072 16245 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43773
I20260925 02:31:43.763167 16245 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:43.763389 16245 heartbeater.cc:511] Master 127.10.13.62:43773 requested a full tablet report, sending...
I20260925 02:31:43.763839 15916 ts_manager.cc:194] Registered new tserver with Master: 1a5df011198d40e2a8d5b455c64625ee (127.10.13.2:40557)
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
I20260925 02:31:43.764298 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:50027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:43.781731 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43773
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:43.864449 16250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:43.864694 16250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:43.864734 16250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:43.865960 16250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:43.866041 16250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:43.866092 16250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:43.867699 16250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:43.867983 16250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:43.868232 16250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:31:43.873956 16250 server_base.cc:1034] running on GCE node
W20260925 02:31:43.873955 16256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:43.874150 16258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:43.873912 16255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:43.874454 16250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:43.874727 16250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:43.876025 16250 hybrid_clock.cc:648] HybridClock initialized: now 1790303503876001 us; error 32 us; skew 500 ppm
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.878867 16250 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:43.879302 16250 webserver.cc:460] Webserver started at http://127.10.13.3:36215/ using document root <none> and password file <none>
I20260925 02:31:43.879495 16250 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:43.879536 16250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:43.879632 16250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:43.880735 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "1c1ba5a5fc5e408688ff41e7b4e732e0"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "8c2440b4cbd366cba5972eb717500b57"
server_key_iv: "0cad9d6941c9021c48f0a618beae5e32"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.881134 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "1c1ba5a5fc5e408688ff41e7b4e732e0"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "8c2440b4cbd366cba5972eb717500b57"
server_key_iv: "0cad9d6941c9021c48f0a618beae5e32"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.882475 16250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:43.883235 16265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.883409 16250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:43.883503 16250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "1c1ba5a5fc5e408688ff41e7b4e732e0"
format_stamp: "Formatted at 2026-09-25 02:31:43 on dist-test-slave-q8mg"
server_key: "8c2440b4cbd366cba5972eb717500b57"
server_key_iv: "0cad9d6941c9021c48f0a618beae5e32"
server_key_version: "encryptionkey@0"
I20260925 02:31:43.883575 16250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:43.894680 16250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:43.897212 16250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:43.897378 16250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:43.897645 16250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:43.897993 16250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:43.898052 16250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.898095 16250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:43.898132 16250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:43.903820 16250 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:36499
I20260925 02:31:43.903903 16379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:36499 every 8 connection(s)
I20260925 02:31:43.904263 16250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:43.906643 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 16250
I20260925 02:31:43.906741 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:43.906947 10292 external_mini_cluster.cc:1428] Setting key a60e6a9ee1f94ce18fbd049d3d7a217d
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:43.915006 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.905180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:46097 (local address 127.10.13.62:43773)
0925 02:31:43.905262 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.905265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.905397 (+ 132us) server_negotiation.cc:407] Connection header received
0925 02:31:43.906303 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.906304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.906314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.906363 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.907366 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.908152 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.909292 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.909510 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.909592 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.911819 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:43.911821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:43.911821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:43.911829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:43.913497 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.914026 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.914029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.914030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.914087 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.914387 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.914387 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.914387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.914496 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:43.914562 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:43.914846 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 02:31:43.914919 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:31:43.915112 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.905097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43773 (local address 127.10.13.3:46097)
0925 02:31:43.905262 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:43.905311 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:43.906209 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:43.906462 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:43.906467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:43.906877 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:43.907252 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:43.907260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.908232 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.908233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:43.909195 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:43.909197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.909597 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.909598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:43.909733 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.910325 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:43.910334 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:43.911707 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:43.913660 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:43.913667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:43.913670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:43.913905 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:43.914195 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:43.914196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:43.914196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:43.914323 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:43.914568 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:43.914573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:43.914697 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 02:31:43.914783 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:31:43.914910 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260925 02:31:43.915665 16380 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43773
I20260925 02:31:43.915755 16380 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:43.915903 16380 heartbeater.cc:511] Master 127.10.13.62:43773 requested a full tablet report, sending...
I20260925 02:31:43.916296 15916 ts_manager.cc:194] Registered new tserver with Master: 1c1ba5a5fc5e408688ff41e7b4e732e0 (127.10.13.3:36499)
I20260925 02:31:43.916703 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:46097
I20260925 02:31:43.919052 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/test-admin.keytab.
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.938134 10292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 02:31:43.940110 16387 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:43.939241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.10.13.1:41177)
0925 02:31:43.939433 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.939438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.939448 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:43.939476 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.939478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.939492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.939561 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.939995 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51048: BlockingRecv error: recv got EOF from 127.0.0.1:51048 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
W20260925 02:31:43.940121 10292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.941212 10292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.942171 10292 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:43.950408 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.942829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.10.13.1:41177)
0925 02:31:43.942895 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.942897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.942905 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:43.942948 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.942949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.942961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.943010 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.943413 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.944145 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.945074 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.945405 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.945464 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.947720 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:43.947728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:43.947728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:43.947735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:43.949381 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.949725 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.949729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.949729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.949783 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.949959 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.949959 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.949960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.950096 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:43.950226 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:43.950266 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 02:31:43.950328 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 02:31:43.952536 16387 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:43.951791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.10.13.1:41177)
0925 02:31:43.951827 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.951829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.951921 (+ 92us) server_negotiation.cc:407] Connection header received
0925 02:31:43.951958 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.951958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.951968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.952017 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.952477 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51062: BlockingRecv error: recv got EOF from 127.0.0.1:51062 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 02:31:43.952672 10292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.953855 10292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.954771 10292 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:43.962016 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.955307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.10.13.1:41177)
0925 02:31:43.955367 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.955369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.955442 (+ 73us) server_negotiation.cc:407] Connection header received
0925 02:31:43.955519 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.955519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.955531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.955603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.955979 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.956456 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.957259 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.957460 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.957549 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.959608 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:43.959610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:43.959610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:43.959618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:43.960819 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.961216 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.961218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.961219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.961282 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.961492 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.961493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.961493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.961601 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:43.961704 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:43.961823 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 02:31:43.961905 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 02:31:43.963919 16387 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:43.963252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.10.13.1:41177)
0925 02:31:43.963309 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.963311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.963369 (+ 58us) server_negotiation.cc:407] Connection header received
0925 02:31:43.963443 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.963443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.963453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.963525 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.963871 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51088: BlockingRecv error: recv got EOF from 127.0.0.1:51088 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 02:31:43.964018 10292 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790303478591324-10292-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.965147 10292 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:43.965977 10292 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:31:43 dist-test-slave-q8mg krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:31:43.972793 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:43.966463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.10.13.1:41177)
0925 02:31:43.966517 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:43.966519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:43.966542 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:31:43.966591 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:43.966592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:43.966600 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:43.966653 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:43.966994 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.967446 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.968137 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:43.968307 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:43.968356 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:43.970445 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:43.970446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:43.970447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:43.970453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:43.971889 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.972214 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.972215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.972216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.972261 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:43.972429 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:43.972429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:43.972429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:43.972515 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:43.972583 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:43.972682 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:31:43.972734 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:31:43.973552 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15978
I20260925 02:31:43.979384 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 16115
I20260925 02:31:43.984313 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 16250
I20260925 02:31:43.988581 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 15883
2026-09-25T02:31:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1887 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:44 dist-test-slave-q8mg krb5kdc[16400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:44 dist-test-slave-q8mg krb5kdc[16400](info): set up 2 sockets
Sep 25 02:31:44 dist-test-slave-q8mg krb5kdc[16400](info): commencing operation
krb5kdc: starting...
W20260925 02:31:45.031450 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.62.keytab.
Entry for principal oryx/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.62.keytab.
I20260925 02:31:45.153560 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39793
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.62.keytab
--principal=oryx/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:45.235520 16413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:45.235720 16413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:45.235742 16413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:45.236897 16413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:45.236936 16413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:45.236951 16413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:45.236963 16413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:45.237022 16413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:45.238452 16413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.62.keytab
--principal=oryx/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:45.238709 16413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:45.238945 16413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:45.244064 16418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:45.244215 16413 server_base.cc:1034] running on GCE node
W20260925 02:31:45.244059 16419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:45.244408 16421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:45.244719 16413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:45.245052 16413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:45.246205 16413 hybrid_clock.cc:648] HybridClock initialized: now 1790303505246183 us; error 38 us; skew 500 ppm
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:45.253681 16413 init.cc:377] Logged in from keytab as oryx/127.10.13.62@KRBTEST.COM (short username oryx)
I20260925 02:31:45.254132 16413 webserver.cc:460] Webserver started at http://127.10.13.62:41723/ using document root <none> and password file <none>
I20260925 02:31:45.254323 16413 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:45.254364 16413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:45.254451 16413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:45.255491 16413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "e1356f4fefc949c1b76552c5092f0dd2"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "2e6bbe4c1a7c389a0f91c3e7704fa4bb"
server_key_iv: "a9143c3f90e24b3b5e9da579e541122b"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.255812 16413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "e1356f4fefc949c1b76552c5092f0dd2"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "2e6bbe4c1a7c389a0f91c3e7704fa4bb"
server_key_iv: "a9143c3f90e24b3b5e9da579e541122b"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.257145 16413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:45.257910 16428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.258116 16413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:45.258213 16413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "e1356f4fefc949c1b76552c5092f0dd2"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "2e6bbe4c1a7c389a0f91c3e7704fa4bb"
server_key_iv: "a9143c3f90e24b3b5e9da579e541122b"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.258268 16413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:45.269196 16413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:45.270948 16413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:45.271059 16413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:45.275336 16413 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:39793
I20260925 02:31:45.275424 16481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:39793 every 8 connection(s)
I20260925 02:31:45.275767 16413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:45.276830 16482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:45.278312 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 16413
I20260925 02:31:45.278434 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:45.278649 10292 external_mini_cluster.cc:1428] Setting key 04419466305612b025bbe9cd5a658e91
I20260925 02:31:45.279613 16482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: Bootstrap starting.
I20260925 02:31:45.280226 16482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:45.280524 16482 log.cc:824] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:45.281266 16482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: No bootstrap required, opened a new log
I20260925 02:31:45.283099 16482 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } }
I20260925 02:31:45.283263 16482 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:45.283314 16482 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1356f4fefc949c1b76552c5092f0dd2, State: Initialized, Role: FOLLOWER
I20260925 02:31:45.283751 16482 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } }
I20260925 02:31:45.283878 16482 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:45.283926 16482 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:45.283985 16482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:45.284530 16482 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } }
I20260925 02:31:45.284662 16482 leader_election.cc:302] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1356f4fefc949c1b76552c5092f0dd2; no voters:
I20260925 02:31:45.284828 16482 leader_election.cc:288] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:45.284902 16487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:45.285112 16487 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 1 LEADER]: Becoming Leader. State: Replica: e1356f4fefc949c1b76552c5092f0dd2, State: Running, Role: LEADER
I20260925 02:31:45.285132 16482 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:45.286674 16487 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } }
I20260925 02:31:45.287366 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1356f4fefc949c1b76552c5092f0dd2. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } }
I20260925 02:31:45.287576 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:45.287451 16489 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } }
I20260925 02:31:45.287847 16489 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
W20260925 02:31:45.288156 16501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:31:45.288234 16501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:31:45.288296 16502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:45.288470 16502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:45.289945 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.279263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.10.13.62:39793)
0925 02:31:45.279469 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:45.279478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:45.279504 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:31:45.279957 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:45.279963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:45.280019 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:45.280176 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:45.280908 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.281857 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.282623 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.283032 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.283177 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.285954 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:45.285959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:45.285960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:45.285973 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:45.288119 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.288579 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.288582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.288583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.288688 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.288952 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.288953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.288953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.289271 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:45.289412 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:45.289659 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 02:31:45.289804 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260925 02:31:45.290340 16502 catalog_manager.cc:1347] Generated new cluster ID: a42275f823954f4c9d9acda7185bab48
I20260925 02:31:45.290391 16502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:45.299604 16502 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:45.300169 16502 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:45.309518 16502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: Generated new TSK 0
I20260925 02:31:45.309724 16502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab.
Entry for principal oryx/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab.
I20260925 02:31:45.344288 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39793
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab
--principal=oryx/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:45.430089 16508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:45.430353 16508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:45.430398 16508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:45.431631 16508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:45.431710 16508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:45.431756 16508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:45.433322 16508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab
--principal=oryx/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:45.433612 16508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:45.433854 16508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:45.439495 16514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:45.439495 16516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:45.439565 16513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:45.439821 16508 server_base.cc:1034] running on GCE node
I20260925 02:31:45.440052 16508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:45.440362 16508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:45.441542 16508 hybrid_clock.cc:648] HybridClock initialized: now 1790303505441529 us; error 31 us; skew 500 ppm
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:45.444195 16508 init.cc:377] Logged in from keytab as oryx/127.10.13.1@KRBTEST.COM (short username oryx)
I20260925 02:31:45.444751 16508 webserver.cc:460] Webserver started at http://127.10.13.1:36207/ using document root <none> and password file <none>
I20260925 02:31:45.445072 16508 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:45.445149 16508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:45.445297 16508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:45.446353 16508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "b243f2481bfe49279f507a3f9421aa35"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "c326d8bf82b86b2d65ff1652c8387a47"
server_key_iv: "068228bfbf7f2a2ea1e4cd643a484f48"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.446723 16508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "b243f2481bfe49279f507a3f9421aa35"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "c326d8bf82b86b2d65ff1652c8387a47"
server_key_iv: "068228bfbf7f2a2ea1e4cd643a484f48"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.448117 16508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:45.448920 16523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.449095 16508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:45.449198 16508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "b243f2481bfe49279f507a3f9421aa35"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "c326d8bf82b86b2d65ff1652c8387a47"
server_key_iv: "068228bfbf7f2a2ea1e4cd643a484f48"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.449277 16508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:45.460493 16508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:45.462702 16508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:45.462869 16508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:45.463085 16508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:45.463414 16508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:45.463449 16508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.463469 16508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:45.463516 16508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.469530 16508 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:38135
I20260925 02:31:45.469609 16637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:38135 every 8 connection(s)
I20260925 02:31:45.469990 16508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:45.479331 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.470792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:59417 (local address 127.10.13.62:39793)
0925 02:31:45.470838 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:45.470840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:45.471060 (+ 220us) server_negotiation.cc:407] Connection header received
0925 02:31:45.471542 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:45.471543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:45.471554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:45.471608 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:45.472552 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.473136 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.474133 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.474333 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.474388 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.476479 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:45.476481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:45.476481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:45.476489 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:45.477800 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.478293 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.478294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.478294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.478360 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.478629 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.478629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.478630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.478752 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:45.478828 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:45.479132 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:31:45.479223 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:31:45.479523 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.470756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:59417)
0925 02:31:45.470928 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:45.470963 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:45.471433 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:45.471677 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:45.471680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:45.472233 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:45.472427 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.472432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.473221 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.473222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.474053 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.474056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.474412 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.474413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.474511 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.474973 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:45.474982 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:45.476376 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:45.477907 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.477916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.477918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.478208 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.478437 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.478438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.478439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.478551 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.478831 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:45.478839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:45.478982 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 02:31:45.479096 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:31:45.479265 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260925 02:31:45.480088 16638 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39793
I20260925 02:31:45.480136 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 16508
I20260925 02:31:45.480180 16638 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:45.480253 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:45.480383 16638 heartbeater.cc:511] Master 127.10.13.62:39793 requested a full tablet report, sending...
I20260925 02:31:45.480499 10292 external_mini_cluster.cc:1428] Setting key e90cf295a89241074fd53c78e212506d
I20260925 02:31:45.481043 16445 ts_manager.cc:194] Registered new tserver with Master: b243f2481bfe49279f507a3f9421aa35 (127.10.13.1:38135)
I20260925 02:31:45.481709 16445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:59417
WARNING: no policy specified for oryx/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab.
Entry for principal oryx/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab.
I20260925 02:31:45.507331 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39793
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab
--principal=oryx/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:45.593616 16643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:45.593840 16643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:45.593868 16643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:45.595150 16643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:45.595204 16643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:45.595232 16643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:45.596803 16643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab
--principal=oryx/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:45.597165 16643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:45.597429 16643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:45.602671 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:45.602710 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:45.602721 16643 server_base.cc:1034] running on GCE node
W20260925 02:31:45.602671 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:45.603155 16643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:45.603422 16643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:45.604607 16643 hybrid_clock.cc:648] HybridClock initialized: now 1790303505604576 us; error 48 us; skew 500 ppm
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:45.607285 16643 init.cc:377] Logged in from keytab as oryx/127.10.13.2@KRBTEST.COM (short username oryx)
I20260925 02:31:45.607787 16643 webserver.cc:460] Webserver started at http://127.10.13.2:33277/ using document root <none> and password file <none>
I20260925 02:31:45.608050 16643 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:45.608127 16643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:45.608285 16643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:45.609479 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "efc1d98e39584e11beb4a6073d74f9a5"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "b708ae743c107c3a3440ab17b14ec0b0"
server_key_iv: "f3f54d6faa6a6be4fec0921377597beb"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.609860 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "efc1d98e39584e11beb4a6073d74f9a5"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "b708ae743c107c3a3440ab17b14ec0b0"
server_key_iv: "f3f54d6faa6a6be4fec0921377597beb"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.611330 16643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:31:45.612238 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.612499 16643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:45.612623 16643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "efc1d98e39584e11beb4a6073d74f9a5"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "b708ae743c107c3a3440ab17b14ec0b0"
server_key_iv: "f3f54d6faa6a6be4fec0921377597beb"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.612709 16643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:45.626717 16643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:45.628691 16643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:45.628978 16643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:45.629240 16643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:45.629571 16643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:45.629627 16643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.629669 16643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:45.629693 16643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.635681 16643 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:41771
I20260925 02:31:45.635735 16772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:41771 every 8 connection(s)
I20260925 02:31:45.636057 16643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:45.642836 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 16643
I20260925 02:31:45.642930 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:45.643141 10292 external_mini_cluster.cc:1428] Setting key 9d22845e163a56101e6a813d9b64ea9a
I20260925 02:31:45.645888 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.636854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:48353 (local address 127.10.13.62:39793)
0925 02:31:45.636978 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:45.636981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:45.637143 (+ 162us) server_negotiation.cc:407] Connection header received
0925 02:31:45.637670 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:45.637671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:45.637682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:45.637753 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:45.638673 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.639191 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.640133 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.640313 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.640378 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.642731 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:45.642733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:45.642733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:45.642740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:45.644266 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.644798 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.644798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.644799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.644849 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.645199 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.645199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.645200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.645317 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:45.645405 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:45.645707 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 02:31:45.645791 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 02:31:45.646150 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.636806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.2:48353)
0925 02:31:45.637012 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:45.637051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:45.637552 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:45.637808 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:45.637812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:45.638383 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:45.638579 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.638584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.639279 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.639280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.640057 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.640059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.640425 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.640427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.640566 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.641180 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:45.641189 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:45.642611 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:45.644353 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.644361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.644364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.644721 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.644973 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.644974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.644974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.645130 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.645398 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:45.645407 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:45.645561 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 02:31:45.645687 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:31:45.645863 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":132,"threads_started":1}
I20260925 02:31:45.646708 16773 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39793
I20260925 02:31:45.646827 16773 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:45.647017 16773 heartbeater.cc:511] Master 127.10.13.62:39793 requested a full tablet report, sending...
I20260925 02:31:45.647482 16445 ts_manager.cc:194] Registered new tserver with Master: efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771)
I20260925 02:31:45.647894 16445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:48353
WARNING: no policy specified for oryx/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab.
Entry for principal oryx/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab.
I20260925 02:31:45.672410 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39793
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab
--principal=oryx/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:45.756891 16778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:45.757215 16778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:45.757261 16778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:45.759320 16778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:45.759438 16778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:45.759505 16778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:45.761991 16778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab
--principal=oryx/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:45.762290 16778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:45.762575 16778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:45.768256 16786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:45.768256 16783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:45.768556 16784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:45.769272 16778 server_base.cc:1034] running on GCE node
I20260925 02:31:45.769484 16778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:45.769750 16778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:45.770936 16778 hybrid_clock.cc:648] HybridClock initialized: now 1790303505770913 us; error 30 us; skew 500 ppm
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:45.773733 16778 init.cc:377] Logged in from keytab as oryx/127.10.13.3@KRBTEST.COM (short username oryx)
I20260925 02:31:45.774185 16778 webserver.cc:460] Webserver started at http://127.10.13.3:45307/ using document root <none> and password file <none>
I20260925 02:31:45.774439 16778 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:45.774487 16778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:45.774643 16778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:45.775766 16778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "b1da9111651a4f6085b2cccaff0a2325"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "ff2a01ce76870a25732286a4938608ef"
server_key_iv: "6957782eb629e96cb52465850e2732d3"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.776129 16778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "b1da9111651a4f6085b2cccaff0a2325"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "ff2a01ce76870a25732286a4938608ef"
server_key_iv: "6957782eb629e96cb52465850e2732d3"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.777609 16778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:45.778458 16793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.778661 16778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:45.778759 16778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "b1da9111651a4f6085b2cccaff0a2325"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "ff2a01ce76870a25732286a4938608ef"
server_key_iv: "6957782eb629e96cb52465850e2732d3"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.778834 16778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:45.800573 16778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:45.803023 16778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:45.803195 16778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:45.803475 16778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:45.803808 16778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:45.803843 16778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.803866 16778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:45.803879 16778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.810077 16778 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:42285
I20260925 02:31:45.810124 16906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:42285 every 8 connection(s)
I20260925 02:31:45.810449 16778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:45.817916 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 16778
I20260925 02:31:45.818017 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:45.818279 10292 external_mini_cluster.cc:1428] Setting key d5002be45cad200f5908ac8eb9ac22c5
I20260925 02:31:45.820082 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.811139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:39121 (local address 127.10.13.62:39793)
0925 02:31:45.811189 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:45.811191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:45.811415 (+ 224us) server_negotiation.cc:407] Connection header received
0925 02:31:45.811959 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:45.811961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:45.811973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:45.812030 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:45.813064 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.813567 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.814620 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.814821 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.814900 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.817232 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:45.817234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:45.817234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:45.817241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:45.818783 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.819257 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.819258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.819258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.819323 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.819556 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.819557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.819557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.819657 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:45.819704 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:45.819954 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 02:31:45.820019 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:31:45.820255 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.811117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:39121)
0925 02:31:45.811279 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:45.811322 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:45.811850 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:45.812104 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:45.812108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:45.812693 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:45.812954 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.812960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.813674 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.813675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.814507 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.814509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.814921 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.814922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.815058 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.815575 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:45.815584 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:45.817142 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:45.818854 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.818861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.818863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.819187 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.819400 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.819400 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.819401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.819513 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.819711 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:45.819715 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:45.819834 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 02:31:45.819918 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:31:45.820038 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260925 02:31:45.820719 16907 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39793
I20260925 02:31:45.820839 16907 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:45.821067 16907 heartbeater.cc:511] Master 127.10.13.62:39793 requested a full tablet report, sending...
I20260925 02:31:45.821439 16445 ts_manager.cc:194] Registered new tserver with Master: b1da9111651a4f6085b2cccaff0a2325 (127.10.13.3:42285)
I20260925 02:31:45.821820 16445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:39121
I20260925 02:31:45.830204 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.61.keytab.
Entry for principal oryx/127.10.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.61.keytab.
I20260925 02:31:45.856592 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.61:45569
--webserver_interface=127.10.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.61.keytab
--principal=oryx/127.10.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39793,127.10.13.61:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:45.941072 16912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:45.941309 16912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:45.941346 16912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:45.942507 16912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:45.942582 16912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:45.942608 16912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:45.942637 16912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:45.942668 16912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:45.944145 16912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39793,127.10.13.61:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.61.keytab
--principal=oryx/127.10.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.61:45569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
W20260925 02:31:45.944295 16912 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.13.62:39793,127.10.13.61:45569'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 02:31:45.944456 16912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:45.944695 16912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:45.949687 16920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:45.949712 16917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:45.949718 16918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:45.949905 16912 server_base.cc:1034] running on GCE node
I20260925 02:31:45.950181 16912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:45.950426 16912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:45.951592 16912 hybrid_clock.cc:648] HybridClock initialized: now 1790303505951568 us; error 27 us; skew 500 ppm
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:45.954397 16912 init.cc:377] Logged in from keytab as oryx/127.10.13.61@KRBTEST.COM (short username oryx)
I20260925 02:31:45.954821 16912 webserver.cc:460] Webserver started at http://127.10.13.61:32915/ using document root <none> and password file <none>
I20260925 02:31:45.955004 16912 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:45.955046 16912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:45.955144 16912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:45.956161 16912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data/instance:
uuid: "ea1da0ece3a1414aa0bb4138754dea89"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "35bc419fd54687296aaa2782096b37e0"
server_key_iv: "85bebe8b4fe88f096b9051520f05dc67"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.956463 16912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal/instance:
uuid: "ea1da0ece3a1414aa0bb4138754dea89"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "35bc419fd54687296aaa2782096b37e0"
server_key_iv: "85bebe8b4fe88f096b9051520f05dc67"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.957857 16912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:45.958606 16927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:45.958786 16912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:45.958901 16912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
uuid: "ea1da0ece3a1414aa0bb4138754dea89"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "35bc419fd54687296aaa2782096b37e0"
server_key_iv: "85bebe8b4fe88f096b9051520f05dc67"
server_key_version: "encryptionkey@0"
I20260925 02:31:45.958981 16912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:45.972522 16912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:45.974671 16912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:45.974843 16912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:45.978839 16912 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.61:45569
I20260925 02:31:45.978873 16979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.61:45569 every 8 connection(s)
I20260925 02:31:45.979242 16912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
I20260925 02:31:45.980232 16980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:45.981995 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 16912
I20260925 02:31:45.982115 16980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.61@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.13.61@KRBTEST.COM
I20260925 02:31:45.993216 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.982910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.10.13.62:39793)
0925 02:31:45.982989 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:45.982992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:45.983213 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:31:45.983684 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:45.983686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:45.983697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:45.983755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:45.984531 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.985419 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.986659 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.986902 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.986971 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.990131 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:45.990133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:45.990133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:45.990141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:45.991787 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.992192 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.992193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.992194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.992257 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.992502 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.992503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.992504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.992659 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:45.992740 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:45.993010 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:31:45.993104 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:31:45.993292 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.982833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.10.13.61:45569)
0925 02:31:45.983023 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:45.983032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:45.983227 (+ 195us) server_negotiation.cc:407] Connection header received
0925 02:31:45.983606 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:45.983616 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:45.983657 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:45.983762 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:45.984538 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.985766 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.986798 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.987205 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.987258 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.990161 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:45.990169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:45.990169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:45.990176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:45.991750 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.992139 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.992142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.992142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.992228 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.992444 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.992445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.992446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.992664 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:45.992834 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:45.992938 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 02:31:45.993081 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260925 02:31:45.993412 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.982857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:36832)
0925 02:31:45.983054 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:45.983095 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:45.983592 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:45.983788 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:45.983791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:45.984173 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:45.984426 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.984433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.985541 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.985542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.986579 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.986581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.987012 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.987013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.987154 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.987695 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:45.987706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:45.990013 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:45.991893 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.991901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.991903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.992109 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.992329 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.992330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.992331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.992428 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.992741 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:45.992749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:45.992818 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 02:31:45.992979 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 02:31:45.993248 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":126,"thread_start_us":113,"threads_started":1}
I20260925 02:31:45.994050 16980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:31:45 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.61@KRBTEST.COM for oryx/127.10.13.61@KRBTEST.COM
I20260925 02:31:45.999910 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.994325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.0.0.1:51218)
0925 02:31:45.994377 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:45.994388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:45.994440 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:45.994527 (+ 87us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:45.994527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:45.994635 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:45.994665 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.994666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.995314 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.995314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.995772 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:45.995773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.996149 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.996150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:45.996190 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.996509 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:45.996513 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:45.997697 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:45.999031 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.999031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.999032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.999209 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.999402 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:45.999403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:45.999403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:45.999453 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:45.999672 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:45.999672 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:45.999707 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:31:45.999766 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:31:45.999841 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 02:31:45.999995 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:45.994351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.10.13.61:45569)
0925 02:31:45.994384 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:45.994386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:45.994424 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:31:45.994477 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:45.994477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:45.994486 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:45.994525 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:45.994726 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.995232 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.995846 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:45.996045 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:45.996131 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:45.997774 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:45.997775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:45.997775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:45.997781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:45.998924 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.999281 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.999281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.999282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.999340 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:45.999512 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:45.999512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:45.999513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:45.999606 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:45.999665 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:45.999782 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 02:31:45.999883 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":16}
I20260925 02:31:46.001411 16980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Bootstrap starting.
I20260925 02:31:46.002287 16980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:46.002576 16980 log.cc:824] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:46.003437 16980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: No bootstrap required, opened a new log
I20260925 02:31:46.004949 16980 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } }
I20260925 02:31:46.005087 16980 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:46.005110 16980 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1da0ece3a1414aa0bb4138754dea89, State: Initialized, Role: FOLLOWER
I20260925 02:31:46.005259 16980 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } }
I20260925 02:31:46.005584 16980 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:46.005542 16986 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } } }
I20260925 02:31:46.005667 16986 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:31:46.007275 16997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:31:46.007342 16997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:31:46.008910 16912 master_runner.cc:428] Detected that this master ea1da0ece3a1414aa0bb4138754dea89 is joining an existing cluster
I20260925 02:31:46.008960 16912 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.13.61:45569
I20260925 02:31:46.016742 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.010010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.10.13.61:45569)
0925 02:31:46.010049 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.010050 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.010142 (+ 92us) server_negotiation.cc:407] Connection header received
0925 02:31:46.010210 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.010211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.010220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.010275 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.010660 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.011393 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.012237 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.012423 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.012501 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.013754 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.013755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.013755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.013762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.015577 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.015912 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.015913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.015913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.015966 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.016187 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.016188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.016188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.016310 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.016429 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.016527 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.016616 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:31:46.017017 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.009952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.0.0.1:51220)
0925 02:31:46.010086 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.010097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.010155 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.010279 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.010280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.010458 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.010565 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.010567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.011483 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.011484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.012155 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.012157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.012515 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.012516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.012578 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.013271 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.013275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.013672 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.015669 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.015669 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.015670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.015839 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.016041 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.016042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.016042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.016108 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.016378 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.016379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.016417 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.016505 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.016670 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260925 02:31:46.017709 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.009947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.10.13.62:39793)
0925 02:31:46.010003 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.010006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.010647 (+ 641us) server_negotiation.cc:407] Connection header received
0925 02:31:46.010738 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.010739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.010752 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.010823 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.011114 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.011727 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.012821 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.013141 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.013196 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.014278 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.014280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.014280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.014287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.016132 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.016590 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.016591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.016591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.016648 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.017014 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.017015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.017016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.017168 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.017214 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.017522 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.017612 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 02:31:46.018026 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.010445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:36842)
0925 02:31:46.010587 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.010598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.010671 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.010844 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.010845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.010989 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.011031 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.011032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.011821 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.011822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.012685 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.012688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.013217 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.013218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.013286 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.013736 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.013741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.014169 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.016225 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.016226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.016226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.016503 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.016721 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.016721 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.016722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.016934 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.017365 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.017366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.017408 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.017825 (+ 417us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.017929 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.62@KRBTEST.COM for oryx/127.10.13.61@KRBTEST.COM
I20260925 02:31:46.027835 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.020301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.10.13.61:45569)
0925 02:31:46.020344 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.020346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.020480 (+ 134us) server_negotiation.cc:407] Connection header received
0925 02:31:46.020576 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.020576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.020584 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.020624 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.021209 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.021641 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.022780 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.022948 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.022996 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.025151 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.025153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.025153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.025161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.026572 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.026980 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.026981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.026981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.027043 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.027270 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.027271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.027271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.027381 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.027460 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.027659 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.027738 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:31:46.027849 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.020220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.0.0.1:51234)
0925 02:31:46.020376 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.020429 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.020502 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.020640 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.020643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.021047 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.021129 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.021131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.021722 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.021723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.022690 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.022693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.023024 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.023025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.023108 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.023527 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.023532 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.025059 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.026683 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.026690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.026693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.026899 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.027127 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.027128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.027128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.027197 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.027460 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.027470 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.027523 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.027633 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.027754 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260925 02:31:46.028427 16445 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.13.61:45569: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89"
member_type: NON_VOTER
last_known_addr {
host: "127.10.13.61"
port: 45569
}
attrs {
promote: true
}
}
dest_uuid: "e1356f4fefc949c1b76552c5092f0dd2"
cas_config_opid_index: -1
I20260925 02:31:46.028767 16445 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } }
I20260925 02:31:46.029315 17007 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ea1da0ece3a1414aa0bb4138754dea89 (127.10.13.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } }
I20260925 02:31:46.029593 16954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:46.030517 17007 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.13.61:45569
I20260925 02:31:46.030545 16954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Refusing update from remote peer e1356f4fefc949c1b76552c5092f0dd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 02:31:46.030622 17007 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } }
I20260925 02:31:46.030690 17007 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:46.030802 16912 master.cc:537] Master@127.10.13.61:45569 shutting down...
I20260925 02:31:46.031044 17008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:31:46.031908 16429 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 02:31:46.032153 16986 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:46.032235 16986 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } }
I20260925 02:31:46.032424 16986 leader_election.cc:288] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793)
I20260925 02:31:46.032620 16456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea1da0ece3a1414aa0bb4138754dea89" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1356f4fefc949c1b76552c5092f0dd2" is_pre_election: true
I20260925 02:31:46.032929 16931 leader_election.cc:302] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: ea1da0ece3a1414aa0bb4138754dea89; no voters: e1356f4fefc949c1b76552c5092f0dd2
I20260925 02:31:46.033058 16986 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:31:46.033972 16912 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:46.034051 16912 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:46.034090 16912 tablet_replica.cc:330] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: stopping tablet replica
I20260925 02:31:46.041671 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.040448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.0.0.1:51236)
0925 02:31:46.041547 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.041626 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1077}
W20260925 02:31:46.041806 16429 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 -> Peer ea1da0ece3a1414aa0bb4138754dea89 (127.10.13.61:45569): Couldn't send request to peer ea1da0ece3a1414aa0bb4138754dea89. Status: Network error: Client connection negotiation failed: client connection to 127.10.13.61:45569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:31:46.046788 16912 master.cc:552] Master@127.10.13.61:45569 shutdown complete.
I20260925 02:31:46.046852 16912 master_runner.cc:305] Clearing existing system tablet
I20260925 02:31:46.046991 16912 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:46.047045 16912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:46.047820 16912 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:46.048393 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.048566 16912 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:46.048629 16912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:46.048656 16912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
uuid: "ea1da0ece3a1414aa0bb4138754dea89"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "35bc419fd54687296aaa2782096b37e0"
server_key_iv: "85bebe8b4fe88f096b9051520f05dc67"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.049538 16912 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:46.051158 16912 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:31:46.051234 16912 log.cc:1196] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 02:31:46.051431 16912 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Deleting consensus metadata
I20260925 02:31:46.051568 16912 master_runner.cc:315] Copying system tablet from 127.10.13.62:39793
I20260925 02:31:46.052219 16912 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: tablet copy: Beginning tablet copy session from remote peer at address 127.10.13.62:39793
I20260925 02:31:46.058988 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.052636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.10.13.62:39793)
0925 02:31:46.052686 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.052688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.052725 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:31:46.052807 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.052808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.052817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.052952 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.053398 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.053897 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.055003 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.055212 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.055274 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.056318 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.056319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.056320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.056326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.057872 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.058275 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.058276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.058276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.058330 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.058535 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.058535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.058535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.058643 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.058695 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.058861 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.058920 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:31:46.059064 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.052517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:36854)
0925 02:31:46.052656 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.052668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.052742 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.052995 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.052996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.053205 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.053275 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.053277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.054005 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.054006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.054904 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.054907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.055294 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.055295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.055365 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.055823 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.055828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.056228 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.057955 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.057956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.057957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.058196 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.058400 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.058401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.058401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.058467 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.058712 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.058713 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.058763 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.058837 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.058950 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":95,"threads_started":1}
I20260925 02:31:46.059438 16466 tablet_copy_service.cc:140] P e1356f4fefc949c1b76552c5092f0dd2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ea1da0ece3a1414aa0bb4138754dea89 ({username='oryx', principal='oryx/127.10.13.61@KRBTEST.COM'} at 127.0.0.1:36854)
I20260925 02:31:46.059543 16466 tablet_copy_service.cc:161] P e1356f4fefc949c1b76552c5092f0dd2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ea1da0ece3a1414aa0bb4138754dea89 at {username='oryx', principal='oryx/127.10.13.61@KRBTEST.COM'} at 127.0.0.1:36854: session id = ea1da0ece3a1414aa0bb4138754dea89-00000000000000000000000000000000
I20260925 02:31:46.060451 16466 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:31:46.061185 16912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:31:46.062335 16912 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: tablet copy: Starting download of 0 data blocks...
I20260925 02:31:46.062513 16912 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: tablet copy: Starting download of 1 WAL segments...
I20260925 02:31:46.063521 16912 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:31:46.064342 16466 tablet_copy_service.cc:342] P e1356f4fefc949c1b76552c5092f0dd2: Request end of tablet copy session ea1da0ece3a1414aa0bb4138754dea89-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.13.61@KRBTEST.COM'} at 127.0.0.1:36854
I20260925 02:31:46.064411 16466 tablet_copy_service.cc:433] P e1356f4fefc949c1b76552c5092f0dd2: ending tablet copy session ea1da0ece3a1414aa0bb4138754dea89-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ea1da0ece3a1414aa0bb4138754dea89
I20260925 02:31:46.067256 16912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:46.068562 17019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.068665 17020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.069922 17022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:46.070101 16912 server_base.cc:1034] running on GCE node
I20260925 02:31:46.070248 16912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:46.070421 16912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:46.071554 16912 hybrid_clock.cc:648] HybridClock initialized: now 1790303506071520 us; error 50 us; skew 500 ppm
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:46.073036 16912 init.cc:377] Logged in from keytab as oryx/127.10.13.61@KRBTEST.COM (short username oryx)
I20260925 02:31:46.073341 16912 webserver.cc:460] Webserver started at http://127.10.13.61:45237/ using document root <none> and password file <none>
I20260925 02:31:46.073487 16912 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:46.073529 16912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:46.074317 16912 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:46.074882 17029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.075023 16912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:46.075073 16912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
uuid: "ea1da0ece3a1414aa0bb4138754dea89"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "35bc419fd54687296aaa2782096b37e0"
server_key_iv: "85bebe8b4fe88f096b9051520f05dc67"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.075133 16912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:46.081319 16912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:46.083344 16912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:46.087132 16912 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.61:45569
I20260925 02:31:46.087528 17081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.61:45569 every 8 connection(s)
I20260925 02:31:46.087623 16912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
I20260925 02:31:46.088570 17082 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:31:46.089138 17082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Bootstrap starting.
I20260925 02:31:46.092020 17082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:31:46.092265 17082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Bootstrap complete.
I20260925 02:31:46.092489 17082 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } }
I20260925 02:31:46.092578 17082 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea1da0ece3a1414aa0bb4138754dea89, State: Initialized, Role: LEARNER
I20260925 02:31:46.092617 17082 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } }
I20260925 02:31:46.092775 17082 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:46.092799 17085 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } }
I20260925 02:31:46.093004 17085 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:31:46.094535 17096 catalog_manager.cc:1259] Loaded cluster ID: a42275f823954f4c9d9acda7185bab48
I20260925 02:31:46.094587 17096 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: loading cluster ID for follower catalog manager: success
I20260925 02:31:46.095579 17096 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: acquiring CA information for follower catalog manager: success
I20260925 02:31:46.095877 17096 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:31:46.103082 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.095256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.10.13.61:45569)
0925 02:31:46.095413 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.095415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.095423 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:46.095447 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.095448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.095614 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.096416 (+ 802us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.096453 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.097414 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.098432 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.098837 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.098915 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.100243 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.100245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.100245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.100253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.101928 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.102303 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.102304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.102305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.102372 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.102563 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.102563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.102564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.102702 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.102807 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.102895 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.102980 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":43,"threads_started":1}
WARNING: no policy specified for oryx/127.10.13.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.60.keytab.
Entry for principal oryx/127.10.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.60.keytab.
I20260925 02:31:46.132565 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.60:33515
--webserver_interface=127.10.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.60.keytab
--principal=oryx/127.10.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39793,127.10.13.61:45569,127.10.13.60:33515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:46.217209 17100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:46.217417 17100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:46.217438 17100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:46.218727 17100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:46.218772 17100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:46.218787 17100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:46.218799 17100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:46.218847 17100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:46.220355 17100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39793,127.10.13.61:45569,127.10.13.60:33515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.60.keytab
--principal=oryx/127.10.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.60:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:46.220628 17100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:46.220913 17100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:46.225874 17106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.225852 17105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.225839 17108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:46.226037 17100 server_base.cc:1034] running on GCE node
I20260925 02:31:46.226318 17100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:46.226574 17100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:46.227708 17100 hybrid_clock.cc:648] HybridClock initialized: now 1790303506227695 us; error 29 us; skew 500 ppm
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:46.230208 17100 init.cc:377] Logged in from keytab as oryx/127.10.13.60@KRBTEST.COM (short username oryx)
I20260925 02:31:46.230710 17100 webserver.cc:460] Webserver started at http://127.10.13.60:41381/ using document root <none> and password file <none>
I20260925 02:31:46.230921 17100 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:46.230962 17100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:46.231062 17100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:46.232087 17100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data/instance:
uuid: "adaf6e52e1f04faf97c0a89bd2975ffe"
format_stamp: "Formatted at 2026-09-25 02:31:46 on dist-test-slave-q8mg"
server_key: "f9b7a53acb3d456e7e0df7a6bc9f2c24"
server_key_iv: "e9f78b301696d777e0ec0903221bf55b"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.232429 17100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal/instance:
uuid: "adaf6e52e1f04faf97c0a89bd2975ffe"
format_stamp: "Formatted at 2026-09-25 02:31:46 on dist-test-slave-q8mg"
server_key: "f9b7a53acb3d456e7e0df7a6bc9f2c24"
server_key_iv: "e9f78b301696d777e0ec0903221bf55b"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.233831 17100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:31:46.234637 17115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.234843 17100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:46.234926 17100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
uuid: "adaf6e52e1f04faf97c0a89bd2975ffe"
format_stamp: "Formatted at 2026-09-25 02:31:46 on dist-test-slave-q8mg"
server_key: "f9b7a53acb3d456e7e0df7a6bc9f2c24"
server_key_iv: "e9f78b301696d777e0ec0903221bf55b"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.234980 17100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:46.249980 17100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:46.251967 17100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:46.252135 17100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:46.256274 17100 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.60:33515
I20260925 02:31:46.256340 17167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.60:33515 every 8 connection(s)
I20260925 02:31:46.256693 17100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
I20260925 02:31:46.257812 17168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:46.258113 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 17100
I20260925 02:31:46.260361 17168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.13.60@KRBTEST.COM
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.60@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:46.269285 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.261272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.10.13.62:39793)
0925 02:31:46.261341 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.261343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.261352 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:46.261399 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.261400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.261410 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.261471 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.261954 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.262483 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.263410 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.263699 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.263757 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.266616 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.266617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.266618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.266636 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.267908 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.268308 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.268310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.268310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.268374 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.268628 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.268629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.268630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.268749 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.268824 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.269111 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.269190 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:31:46.269553 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.261031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:36868)
0925 02:31:46.261201 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.261239 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.261311 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.261494 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.261496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.261787 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.261869 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.261871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.262607 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.262608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.263320 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.263323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.263808 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.263810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.263885 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.264306 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.264318 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.266525 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.268018 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.268025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.268028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.268228 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.268453 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.268454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.268455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.268546 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.268826 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.268834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.268986 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.269087 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.269249 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":120,"threads_started":1}
I20260925 02:31:46.269556 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.258799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.10.13.60:33515)
0925 02:31:46.258981 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.258989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.259013 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:31:46.259429 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.259436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.259482 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.259590 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.260388 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.261702 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.262554 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.262979 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.263073 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.266198 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.266206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.266206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.266213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.267969 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.268405 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.268408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.268409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.268494 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.268705 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.268706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.268706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.268988 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.269151 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.269236 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.269311 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260925 02:31:46.270095 17168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.60@KRBTEST.COM for oryx/127.10.13.61@KRBTEST.COM
I20260925 02:31:46.276537 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.270451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.10.13.61:45569)
0925 02:31:46.270523 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.270525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.270533 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:46.270566 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.270567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.270579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.270647 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.270909 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.271401 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.272088 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.272328 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.272407 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.274042 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.274044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.274044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.274083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.275432 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.275815 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.275816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.275816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.275866 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.276074 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.276074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.276074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.276211 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.276274 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.276407 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.276474 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:31:46.276569 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.270363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.0.0.1:51262)
0925 02:31:46.270395 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.270406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.270440 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.270650 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.270651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.270785 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.270829 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.270831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.271488 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.271489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.271995 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.271997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.272403 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.272404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.272472 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.272808 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.272812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.273951 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.275513 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.275514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.275514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.275743 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.275933 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.275934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.275935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.275997 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.276268 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.276269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.276313 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.276389 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.276476 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 02:31:46.276984 17168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.60@KRBTEST.COM for oryx/127.10.13.60@KRBTEST.COM
I20260925 02:31:46.283285 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.277274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.0.0.1:39502)
0925 02:31:46.277326 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.277337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.277374 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.277509 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.277509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.277615 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.277642 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.277643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.278453 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.278454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.278918 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.278920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.279290 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.279291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.279337 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.279739 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.279744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.280817 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.282224 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.282225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.282226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.282468 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.282667 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.282668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.282668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.282722 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.282969 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.282970 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.283005 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.283073 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.283174 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 02:31:46.283330 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.277328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.10.13.60:33515)
0925 02:31:46.277366 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.277368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.277375 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:31:46.277433 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.277434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.277442 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.277507 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.277710 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.278350 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.279010 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.279209 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.279280 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.280932 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.280933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.280933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.280940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.282141 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.282545 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.282546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.282547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.282600 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.282793 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.282794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.282795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.282899 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.282977 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.283107 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.283223 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":20}
I20260925 02:31:46.284763 17168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Bootstrap starting.
I20260925 02:31:46.285790 17168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:46.286119 17168 log.cc:824] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:46.286939 17168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: No bootstrap required, opened a new log
I20260925 02:31:46.288389 17168 raft_consensus.cc:348] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } }
I20260925 02:31:46.288535 17168 raft_consensus.cc:374] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:46.288559 17168 raft_consensus.cc:729] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaf6e52e1f04faf97c0a89bd2975ffe, State: Initialized, Role: FOLLOWER
I20260925 02:31:46.288705 17168 consensus_queue.cc:260] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } }
I20260925 02:31:46.288995 17173 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } } }
I20260925 02:31:46.289074 17168 sys_catalog.cc:564] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:46.289088 17173 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:31:46.290766 17184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:31:46.290822 17184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:31:46.294472 17100 master_runner.cc:428] Detected that this master adaf6e52e1f04faf97c0a89bd2975ffe is joining an existing cluster
I20260925 02:31:46.294529 17100 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.13.60:33515
I20260925 02:31:46.302965 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.295883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.0.0.1:39504)
0925 02:31:46.296082 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.296091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.296160 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.296309 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.296309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.296642 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.296715 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.296717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.297645 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.297645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.298520 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.298523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.298979 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.298979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.299029 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.299951 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.299955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.300308 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.302025 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.302026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.302026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.302198 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.302405 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.302405 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.302405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.302457 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.302726 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.302726 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.302761 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.302810 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.302887 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260925 02:31:46.303061 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.295974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.10.13.60:33515)
0925 02:31:46.296009 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.296011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.296132 (+ 121us) server_negotiation.cc:407] Connection header received
0925 02:31:46.296223 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.296224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.296232 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.296258 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.296801 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.297551 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.298608 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.298868 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.298904 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.300412 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.300414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.300414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.300421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.301944 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.302273 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.302274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.302275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.302335 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.302522 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.302523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.302524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.302661 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.302729 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.302833 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.302946 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":19}
I20260925 02:31:46.321326 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 16508
I20260925 02:31:46.325861 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.295679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.10.13.62:39793)
0925 02:31:46.295731 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.295734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.296053 (+ 319us) server_negotiation.cc:407] Connection header received
0925 02:31:46.296164 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.296165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.296177 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.296206 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.296751 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.297547 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.298794 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.299085 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.299126 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.322265 (+ 23139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.322268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.322268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.322278 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.324525 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.324968 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.324970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.324970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.325030 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.325247 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.325247 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.325248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.325400 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.325447 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.325681 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.325760 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:46.325872 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.295839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:36870)
0925 02:31:46.295988 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.295998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.296089 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.296274 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.296275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.296575 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.296655 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.296658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.297755 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.297756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.298692 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.298695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.299207 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.299207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.299269 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.321652 (+ 22383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.321658 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.322137 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.324628 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.324629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.324630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.324841 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.325116 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.325117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.325117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.325180 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.325497 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.325498 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.325552 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.325646 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.325762 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260925 02:31:46.328070 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.295738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.10.13.61:45569)
0925 02:31:46.295783 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.295785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.296051 (+ 266us) server_negotiation.cc:407] Connection header received
0925 02:31:46.296173 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.296174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.296185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.296218 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.296807 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.297562 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.298717 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.299009 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.299047 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.324744 (+ 25697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.324746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.324746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.324756 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.326831 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.327200 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.327201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.327201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.327261 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.327481 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.327482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.327483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.327624 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.327688 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.327903 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.327981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:31:46.328097 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.295670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.0.0.1:51274)
0925 02:31:46.295984 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.295996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.296090 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.296290 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.296291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.296646 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.296715 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.296717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.297645 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.297645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.298637 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.298639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.299121 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.299122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.299191 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.324216 (+ 25025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.324221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.324620 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.326917 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.326918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.326918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.327114 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.327336 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.327337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.327337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.327401 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.327715 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.327716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.327774 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.327870 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.327978 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":38,"thread_start_us":80,"threads_started":1}
I20260925 02:31:46.329711 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:38135
--local_ip_for_outbound_sockets=127.10.13.1
--tserver_master_addrs=127.10.13.62:39793
--webserver_port=36207
--webserver_interface=127.10.13.1
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab
--principal=oryx/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569,127.10.13.60:33515 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:31:46.331431 17173 raft_consensus.cc:482] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:46.331517 17173 raft_consensus.cc:504] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } }
I20260925 02:31:46.331782 17173 leader_election.cc:288] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793), ea1da0ece3a1414aa0bb4138754dea89 (127.10.13.61:45569)
I20260925 02:31:46.332024 16456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1356f4fefc949c1b76552c5092f0dd2" is_pre_election: true
I20260925 02:31:46.332118 17056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1da0ece3a1414aa0bb4138754dea89" is_pre_election: true
I20260925 02:31:46.332219 17056 raft_consensus.cc:1799] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 LEARNER]: Handling vote request from an unknown peer adaf6e52e1f04faf97c0a89bd2975ffe
I20260925 02:31:46.332267 17056 raft_consensus.cc:2396] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate adaf6e52e1f04faf97c0a89bd2975ffe for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260925 02:31:46.332438 17116 leader_election.cc:302] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: adaf6e52e1f04faf97c0a89bd2975ffe; no voters: e1356f4fefc949c1b76552c5092f0dd2, ea1da0ece3a1414aa0bb4138754dea89
I20260925 02:31:46.332769 17173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303505, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.62@KRBTEST.COM for oryx/127.10.13.60@KRBTEST.COM
I20260925 02:31:46.333894 17173 raft_consensus.cc:2726] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 02:31:46.336520 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.328993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.10.13.60:33515)
0925 02:31:46.329034 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.329036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.329084 (+ 48us) server_negotiation.cc:407] Connection header received
0925 02:31:46.329168 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.329168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.329178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.329226 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.329559 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.330288 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.331344 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.331612 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.331678 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.333971 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.333972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.333972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.333979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.335375 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.335736 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.335737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.335738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.335797 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.336006 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.336007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.336008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.336117 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.336191 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.336366 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.336432 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:31:46.336525 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.328957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.0.0.1:39508)
0925 02:31:46.329036 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.329046 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.329099 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.329226 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.329227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.329408 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.329462 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.329464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.330379 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.330380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.331242 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.331244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.331734 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.331736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.331804 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.332319 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.332324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.333850 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.335483 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.335484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.335485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.335659 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.335873 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.335874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.335875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.335940 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.336197 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.336198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.336242 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.336327 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.336434 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:31:46.337363 16445 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.13.60:33515: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe"
member_type: NON_VOTER
last_known_addr {
host: "127.10.13.60"
port: 33515
}
attrs {
promote: true
}
}
dest_uuid: "e1356f4fefc949c1b76552c5092f0dd2"
cas_config_opid_index: 5
I20260925 02:31:46.337634 16445 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } }
I20260925 02:31:46.338030 17008 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.338985 17008 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.13.60:33515
I20260925 02:31:46.339068 17008 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.339140 17008 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:46.339401 17100 master.cc:537] Master@127.10.13.60:33515 shutting down...
W20260925 02:31:46.340680 16429 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 -> Peer adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60:33515): Couldn't send request to peer adaf6e52e1f04faf97c0a89bd2975ffe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:31:46.343873 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.339444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.0.0.1:51280)
0925 02:31:46.340289 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.340301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.340345 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.341189 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.341190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.341192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:46.341242 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.341243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.342073 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.342074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.342841 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.342843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.343595 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.343596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.343659 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.343687 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.343740 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.343777 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825}
I20260925 02:31:46.344295 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.340151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.10.13.61:45569)
0925 02:31:46.340844 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.340846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.340855 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:46.340901 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.340902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.340925 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.340989 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:46.341322 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.341997 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.342950 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.343486 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.343593 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.343670 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.344179 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.344226 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667}
I20260925 02:31:46.344501 17056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 LEARNER]: Refusing update from remote peer e1356f4fefc949c1b76552c5092f0dd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:31:46.344702 17007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 02:31:46.344755 17100 raft_consensus.cc:2229] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:46.344831 17100 raft_consensus.cc:2258] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:46.345006 17100 tablet_replica.cc:330] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: stopping tablet replica
I20260925 02:31:46.345299 17056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.346930 17085 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1356f4fefc949c1b76552c5092f0dd2. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.347035 17085 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:31:46.347078 17085 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: NON_VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: true } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.347131 17085 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:31:46.353718 17007 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: attempting to promote NON_VOTER ea1da0ece3a1414aa0bb4138754dea89 to VOTER
I20260925 02:31:46.354015 17007 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } }
I20260925 02:31:46.354588 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.354458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.0.0.1:39518)
0925 02:31:46.354516 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.354554 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:31:46.354753 17056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 LEARNER]: Refusing update from remote peer e1356f4fefc949c1b76552c5092f0dd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:31:46.354969 17008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 02:31:46.355242 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.355156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.0.0.1:39530)
0925 02:31:46.355188 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.355216 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 02:31:46.355415 16429 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 -> Peer adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60:33515): Couldn't send request to peer adaf6e52e1f04faf97c0a89bd2975ffe. Status: Network error: Client connection negotiation failed: client connection to 127.10.13.60:33515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:31:46.355634 17006 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ea1da0ece3a1414aa0bb4138754dea89 (127.10.13.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.355798 17056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ea1da0ece3a1414aa0bb4138754dea89 (127.10.13.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.356613 17006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.356695 17006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:46.356801 17006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.356853 17006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:46.356838 17195 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.356956 17195 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:31:46.358503 17100 master.cc:552] Master@127.10.13.60:33515 shutdown complete.
I20260925 02:31:46.358575 17100 master_runner.cc:305] Clearing existing system tablet
I20260925 02:31:46.358711 17100 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:46.358773 17100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:46.359397 17100 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:46.359866 17198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.360026 17100 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:46.360092 17100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.360123 17100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
uuid: "adaf6e52e1f04faf97c0a89bd2975ffe"
format_stamp: "Formatted at 2026-09-25 02:31:46 on dist-test-slave-q8mg"
server_key: "f9b7a53acb3d456e7e0df7a6bc9f2c24"
server_key_iv: "e9f78b301696d777e0ec0903221bf55b"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.360842 17100 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:46.362419 17100 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:31:46.362608 17100 log.cc:1196] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 02:31:46.362797 17100 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Deleting consensus metadata
I20260925 02:31:46.362941 17100 master_runner.cc:315] Copying system tablet from 127.10.13.62:39793
I20260925 02:31:46.363590 17100 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: tablet copy: Beginning tablet copy session from remote peer at address 127.10.13.62:39793
I20260925 02:31:46.370110 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.363955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.10.13.62:39793)
0925 02:31:46.364002 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.364005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.364135 (+ 130us) server_negotiation.cc:407] Connection header received
0925 02:31:46.364231 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.364232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.364243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.364294 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.364617 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.365353 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.366095 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.366304 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.366386 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.367370 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.367371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.367372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.367379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.368804 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.369274 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.369276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.369276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.369341 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.369553 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.369554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.369554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.369671 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.369748 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.369930 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.370006 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:31:46.370127 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.363919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:36878)
0925 02:31:46.364048 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.364077 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.364160 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.364297 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.364297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.364486 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.364527 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.364529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.365449 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.365450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.365998 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.366000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.366385 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.366385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.366450 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.366960 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.366966 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.367272 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.369018 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.369018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.369019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.369185 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.369412 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.369413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.369414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.369474 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.369753 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.369755 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.369802 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.369896 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.370007 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260925 02:31:46.370702 16466 tablet_copy_service.cc:140] P e1356f4fefc949c1b76552c5092f0dd2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer adaf6e52e1f04faf97c0a89bd2975ffe ({username='oryx', principal='oryx/127.10.13.60@KRBTEST.COM'} at 127.0.0.1:36878)
I20260925 02:31:46.370786 16466 tablet_copy_service.cc:161] P e1356f4fefc949c1b76552c5092f0dd2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer adaf6e52e1f04faf97c0a89bd2975ffe at {username='oryx', principal='oryx/127.10.13.60@KRBTEST.COM'} at 127.0.0.1:36878: session id = adaf6e52e1f04faf97c0a89bd2975ffe-00000000000000000000000000000000
I20260925 02:31:46.371273 16466 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:31:46.371758 17100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:31:46.372747 17100 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: tablet copy: Starting download of 0 data blocks...
I20260925 02:31:46.372942 17100 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: tablet copy: Starting download of 1 WAL segments...
I20260925 02:31:46.373629 17100 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:31:46.374441 16466 tablet_copy_service.cc:342] P e1356f4fefc949c1b76552c5092f0dd2: Request end of tablet copy session adaf6e52e1f04faf97c0a89bd2975ffe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.13.60@KRBTEST.COM'} at 127.0.0.1:36878
I20260925 02:31:46.374498 16466 tablet_copy_service.cc:433] P e1356f4fefc949c1b76552c5092f0dd2: ending tablet copy session adaf6e52e1f04faf97c0a89bd2975ffe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer adaf6e52e1f04faf97c0a89bd2975ffe
I20260925 02:31:46.377210 17100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:46.378646 17205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:46.378820 17100 server_base.cc:1034] running on GCE node
W20260925 02:31:46.378680 17208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.381013 17206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:46.381179 17100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:46.381389 17100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:46.382522 17100 hybrid_clock.cc:648] HybridClock initialized: now 1790303506382482 us; error 61 us; skew 500 ppm
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:46.383985 17100 init.cc:377] Logged in from keytab as oryx/127.10.13.60@KRBTEST.COM (short username oryx)
I20260925 02:31:46.384305 17100 webserver.cc:460] Webserver started at http://127.10.13.60:44349/ using document root <none> and password file <none>
I20260925 02:31:46.384442 17100 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:46.384478 17100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:46.385212 17100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:46.385825 17215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.386001 17100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.386049 17100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
uuid: "adaf6e52e1f04faf97c0a89bd2975ffe"
format_stamp: "Formatted at 2026-09-25 02:31:46 on dist-test-slave-q8mg"
server_key: "f9b7a53acb3d456e7e0df7a6bc9f2c24"
server_key_iv: "e9f78b301696d777e0ec0903221bf55b"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.386116 17100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:46.394587 17100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:46.396405 17100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:46.399649 17100 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.60:33515
I20260925 02:31:46.399971 17100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
I20260925 02:31:46.400650 17268 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:31:46.401198 17268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Bootstrap starting.
I20260925 02:31:46.403446 17268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:31:46.403743 17268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: Bootstrap complete.
I20260925 02:31:46.403965 17268 raft_consensus.cc:348] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } }
I20260925 02:31:46.404066 17268 raft_consensus.cc:729] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: adaf6e52e1f04faf97c0a89bd2975ffe, State: Initialized, Role: LEARNER
I20260925 02:31:46.404110 17268 consensus_queue.cc:260] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } }
I20260925 02:31:46.404273 17268 sys_catalog.cc:564] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:46.404270 17271 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.404489 17271 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: This master's current role is: LEARNER
I20260925 02:31:46.404959 17267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.60:33515 every 8 connection(s)
I20260925 02:31:46.406539 17282 catalog_manager.cc:1259] Loaded cluster ID: a42275f823954f4c9d9acda7185bab48
I20260925 02:31:46.406602 17282 catalog_manager.cc:1552] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: loading cluster ID for follower catalog manager: success
I20260925 02:31:46.407366 17282 catalog_manager.cc:1574] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: acquiring CA information for follower catalog manager: success
I20260925 02:31:46.407665 17282 catalog_manager.cc:1602] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:31:46.454623 17193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:46.454910 17193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:46.454947 17193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:46.456715 17193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:46.456777 17193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:46.456817 17193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:46.459036 17193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab
--principal=oryx/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:38135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=36207
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569,127.10.13.60:33515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:46.459301 17193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:46.459565 17193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:46.465137 17288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.465085 17290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.465112 17287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:46.465837 17193 server_base.cc:1034] running on GCE node
I20260925 02:31:46.466042 17193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:46.466315 17193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:46.467492 17193 hybrid_clock.cc:648] HybridClock initialized: now 1790303506467476 us; error 32 us; skew 500 ppm
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:46.470266 17193 init.cc:377] Logged in from keytab as oryx/127.10.13.1@KRBTEST.COM (short username oryx)
I20260925 02:31:46.470767 17193 webserver.cc:460] Webserver started at http://127.10.13.1:36207/ using document root <none> and password file <none>
I20260925 02:31:46.471025 17193 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:46.471127 17193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:46.472656 17193 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:46.473419 17297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.473640 17193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:31:46.473765 17193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "b243f2481bfe49279f507a3f9421aa35"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "c326d8bf82b86b2d65ff1652c8387a47"
server_key_iv: "068228bfbf7f2a2ea1e4cd643a484f48"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.474224 17193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:46.509625 17193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:46.512326 17193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:46.512518 17193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:46.512814 17193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:46.513186 17193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:46.513248 17193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.513317 17193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:46.513346 17193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.520445 17193 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:38135
I20260925 02:31:46.520509 17410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:38135 every 8 connection(s)
I20260925 02:31:46.520887 17193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:46.526595 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 17193
I20260925 02:31:46.526777 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 16643
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.60@KRBTEST.COM
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.61@KRBTEST.COM
I20260925 02:31:46.533717 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:41771
--local_ip_for_outbound_sockets=127.10.13.2
--tserver_master_addrs=127.10.13.62:39793
--webserver_port=33277
--webserver_interface=127.10.13.2
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab
--principal=oryx/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569,127.10.13.60:33515 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:31:46.535175 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.522093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:54417 (local address 127.10.13.61:45569)
0925 02:31:46.522138 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.522140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.522382 (+ 242us) server_negotiation.cc:407] Connection header received
0925 02:31:46.522890 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.522892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.522904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.522972 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.523722 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.524225 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.525365 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.525610 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.525679 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.532099 (+ 6420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.532100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.532101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.532109 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.533849 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.534316 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.534317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.534317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.534382 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.534584 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.534585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.534586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.534734 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.534779 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.534998 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.535085 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:31:46.535255 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.521702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:44513 (local address 127.10.13.62:39793)
0925 02:31:46.521874 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.521877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.522221 (+ 344us) server_negotiation.cc:407] Connection header received
0925 02:31:46.522849 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.522851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.522865 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.522922 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.523740 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.524511 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.525607 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.525930 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.525992 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.531791 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.531793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.531793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.531802 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.533824 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.534353 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.534354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.534354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.534417 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.534621 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.534622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.534622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.534771 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.534816 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.535021 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.535111 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260925 02:31:46.535372 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.521701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.10.13.1:55403)
0925 02:31:46.521967 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.522011 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.522731 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.523062 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.523063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.523432 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.523619 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.523632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.524844 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.524844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.525667 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.525669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.526172 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.526173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.526260 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.526686 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.526690 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.530100 (+ 3410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.532146 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.532154 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.532157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.532484 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.533063 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.533065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.533065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.533250 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.534611 (+ 1361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.534619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.534792 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.534929 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.535104 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":86,"threads_started":1}
I20260925 02:31:46.535606 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.521701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:44513)
0925 02:31:46.522158 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.522167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.522738 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.523009 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.523013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.523598 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.523649 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.523651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.524602 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.524603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.525518 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.525521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.526045 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.526047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.526112 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.526530 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.526536 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.531687 (+ 5151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.533943 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.533944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.533944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.534262 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.534488 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.534488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.534489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.534551 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.534864 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.534865 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.534911 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.535347 (+ 436us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.535468 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":26,"thread_start_us":104,"threads_started":1}
I20260925 02:31:46.535815 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.521763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:55403 (local address 127.10.13.60:33515)
0925 02:31:46.521903 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.521906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.522089 (+ 183us) server_negotiation.cc:407] Connection header received
0925 02:31:46.522942 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.522944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.522958 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.523012 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.523737 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.524758 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.525783 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.526089 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.526174 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.530228 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.530230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.530230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.530238 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.532039 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.532914 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.532916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.532916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.532981 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.534362 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.534363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.534363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.534521 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.534584 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.535626 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.535721 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260925 02:31:46.536072 17412 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39793
I20260925 02:31:46.536168 17411 heartbeater.cc:347] Connected to a master server at 127.10.13.60:33515
I20260925 02:31:46.536183 17412 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.536249 17411 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.536434 17412 heartbeater.cc:511] Master 127.10.13.62:39793 requested a full tablet report, sending...
I20260925 02:31:46.536437 17411 heartbeater.cc:511] Master 127.10.13.60:33515 requested a full tablet report, sending...
I20260925 02:31:46.536446 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.522010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.10.13.1:54417)
0925 02:31:46.522316 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.522324 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.522744 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.523044 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.523045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.523412 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.523617 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.523621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.524342 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.524343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.525270 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.525273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.525684 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.525685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.525822 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.526463 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.526474 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.532007 (+ 5533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.533992 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.533993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.533994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.534233 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.534456 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.534457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.534457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.534510 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.534828 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.534829 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.534886 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.536122 (+ 1236us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.536355 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":69,"threads_started":1}
I20260925 02:31:46.536901 16445 ts_manager.cc:194] Re-registered known tserver with Master: b243f2481bfe49279f507a3f9421aa35 (127.10.13.1:38135)
I20260925 02:31:46.537098 17413 heartbeater.cc:347] Connected to a master server at 127.10.13.61:45569
I20260925 02:31:46.537159 17413 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.537240 17413 heartbeater.cc:511] Master 127.10.13.61:45569 requested a full tablet report, sending...
I20260925 02:31:46.537475 16445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:44513
I20260925 02:31:46.537629 17046 ts_manager.cc:194] Registered new tserver with Master: b243f2481bfe49279f507a3f9421aa35 (127.10.13.1:38135)
I20260925 02:31:46.537786 17232 ts_manager.cc:194] Registered new tserver with Master: b243f2481bfe49279f507a3f9421aa35 (127.10.13.1:38135)
W20260925 02:31:46.622383 17419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:46.622581 17419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:46.622605 17419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:46.623750 17419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:46.623793 17419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:46.623819 17419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:46.625339 17419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab
--principal=oryx/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:41771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=33277
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569,127.10.13.60:33515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:46.625548 17419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:46.625756 17419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:46.630865 17424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.630865 17427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:46.631080 17419 server_base.cc:1034] running on GCE node
W20260925 02:31:46.630859 17425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:46.631418 17419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:46.631630 17419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:46.632771 17419 hybrid_clock.cc:648] HybridClock initialized: now 1790303506632759 us; error 28 us; skew 500 ppm
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:46.635459 17419 init.cc:377] Logged in from keytab as oryx/127.10.13.2@KRBTEST.COM (short username oryx)
I20260925 02:31:46.635900 17419 webserver.cc:460] Webserver started at http://127.10.13.2:33277/ using document root <none> and password file <none>
I20260925 02:31:46.636154 17419 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:46.636211 17419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:46.637681 17419 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:46.638315 17434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.638518 17419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:46.638592 17419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "efc1d98e39584e11beb4a6073d74f9a5"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "b708ae743c107c3a3440ab17b14ec0b0"
server_key_iv: "f3f54d6faa6a6be4fec0921377597beb"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.638943 17419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:46.674005 17419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:46.676363 17419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:46.676522 17419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:46.676810 17419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:46.677192 17419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:46.677250 17419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.677294 17419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:46.677321 17419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.683935 17419 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:41771
I20260925 02:31:46.684008 17547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:41771 every 8 connection(s)
I20260925 02:31:46.684412 17419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:46.690877 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 17419
I20260925 02:31:46.691054 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 16778
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.60@KRBTEST.COM
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.61@KRBTEST.COM
I20260925 02:31:46.696869 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.685740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:41205 (local address 127.10.13.61:45569)
0925 02:31:46.685784 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.685786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.685900 (+ 114us) server_negotiation.cc:407] Connection header received
0925 02:31:46.687139 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.687140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.687152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.687208 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.687901 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.688749 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.689874 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.690167 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.690207 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.693803 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.693804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.693804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.693812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.695491 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.695876 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.695877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.695877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.695928 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.696209 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.696210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.696210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.696327 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.696386 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.696693 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.696769 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:31:46.696911 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.685180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:33979 (local address 127.10.13.60:33515)
0925 02:31:46.685232 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.685234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.685682 (+ 448us) server_negotiation.cc:407] Connection header received
0925 02:31:46.686997 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.686999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.687014 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.687076 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.687903 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.688706 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.689875 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.690200 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.690239 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.693357 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.693359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.693359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.693367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.695228 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.695733 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.695734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.695735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.695792 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.696173 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.696174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.696174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.696315 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.696386 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.696707 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.696797 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:31:46.697326 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.685283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.10.13.2:33979)
0925 02:31:46.685487 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.685527 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.686801 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.687093 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.687094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.687597 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.687788 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.687794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.688811 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.688812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.689772 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.689774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.690307 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.690307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.690359 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.690791 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.690796 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.693246 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.695326 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.695334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.695336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.695653 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.695998 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.695999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.695999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.696101 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.696394 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.696401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.696561 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.697022 (+ 461us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.697132 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":44,"thread_start_us":129,"threads_started":1}
I20260925 02:31:46.697419 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.685639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.10.13.2:41205)
0925 02:31:46.685846 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.685855 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.686812 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.687227 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.687228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.687590 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.687788 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.687794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.688849 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.688850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.689777 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.689778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.690285 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.690286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.690349 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.691552 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.691557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.693723 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.695585 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.695587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.695587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.695791 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.696002 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.696003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.696003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.696117 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.696416 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.696417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.696556 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.696673 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.696838 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":25,"thread_start_us":127,"threads_started":1}
I20260925 02:31:46.697860 17548 heartbeater.cc:347] Connected to a master server at 127.10.13.60:33515
I20260925 02:31:46.697863 17550 heartbeater.cc:347] Connected to a master server at 127.10.13.61:45569
I20260925 02:31:46.698012 17550 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.698012 17548 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.698235 17550 heartbeater.cc:511] Master 127.10.13.61:45569 requested a full tablet report, sending...
I20260925 02:31:46.698239 17548 heartbeater.cc:511] Master 127.10.13.60:33515 requested a full tablet report, sending...
I20260925 02:31:46.698809 17046 ts_manager.cc:194] Registered new tserver with Master: efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771)
I20260925 02:31:46.698822 17232 ts_manager.cc:194] Registered new tserver with Master: efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771)
I20260925 02:31:46.698822 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:42285
--local_ip_for_outbound_sockets=127.10.13.3
--tserver_master_addrs=127.10.13.62:39793
--webserver_port=45307
--webserver_interface=127.10.13.3
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab
--principal=oryx/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569,127.10.13.60:33515 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:46.702741 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.685281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.2:60249)
0925 02:31:46.685483 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.685526 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.686799 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.687059 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.687063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.687570 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.687788 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.687793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.688527 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.688528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.689428 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.689431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.689853 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.689855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.689989 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.690563 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.690574 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.700080 (+ 9506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.701787 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.701788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.701788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.701948 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.702169 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.702169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.702169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.702220 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.702505 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.702506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.702542 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.702591 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.702669 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":118,"threads_started":1}
I20260925 02:31:46.702845 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.685214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:60249 (local address 127.10.13.62:39793)
0925 02:31:46.685267 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.685269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.685611 (+ 342us) server_negotiation.cc:407] Connection header received
0925 02:31:46.686915 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.686917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.686931 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.686989 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.687898 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.688410 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.689534 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.689750 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.689841 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.700204 (+ 10363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.700207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.700207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.700217 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.701707 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.702020 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.702021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.702022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.702093 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.702284 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.702285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.702286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.702440 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.702511 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.702620 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.702706 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:31:46.703341 17549 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39793
I20260925 02:31:46.703416 17549 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.703505 17549 heartbeater.cc:511] Master 127.10.13.62:39793 requested a full tablet report, sending...
I20260925 02:31:46.703784 16445 ts_manager.cc:194] Re-registered known tserver with Master: efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771)
I20260925 02:31:46.704350 16445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:60249
W20260925 02:31:46.783263 17555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:46.783473 17555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:46.783499 17555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:46.784699 17555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:46.784739 17555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:46.784796 17555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:46.786541 17555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab
--principal=oryx/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:42285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=45307
--tserver_master_addrs=127.10.13.62:39793,127.10.13.61:45569,127.10.13.60:33515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:46.786816 17555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:46.787070 17555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:46.792261 17563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.792287 17561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:46.792287 17560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:46.792303 17555 server_base.cc:1034] running on GCE node
I20260925 02:31:46.792693 17555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:46.793021 17555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:46.794169 17555 hybrid_clock.cc:648] HybridClock initialized: now 1790303506794153 us; error 37 us; skew 500 ppm
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:46.796792 17555 init.cc:377] Logged in from keytab as oryx/127.10.13.3@KRBTEST.COM (short username oryx)
I20260925 02:31:46.797291 17555 webserver.cc:460] Webserver started at http://127.10.13.3:45307/ using document root <none> and password file <none>
I20260925 02:31:46.797554 17555 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:46.797611 17555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:46.798987 17555 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:46.799543 17570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.799741 17555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.799832 17555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "b1da9111651a4f6085b2cccaff0a2325"
format_stamp: "Formatted at 2026-09-25 02:31:45 on dist-test-slave-q8mg"
server_key: "ff2a01ce76870a25732286a4938608ef"
server_key_iv: "6957782eb629e96cb52465850e2732d3"
server_key_version: "encryptionkey@0"
I20260925 02:31:46.800186 17555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:46.812052 17555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:46.814491 17555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:46.814666 17555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:46.814917 17555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:46.815253 17555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:46.815308 17555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.815349 17555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:46.815378 17555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:46.821396 17555 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:42285
I20260925 02:31:46.821481 17683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:42285 every 8 connection(s)
I20260925 02:31:46.821823 17555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:46.825133 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 17555
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.61@KRBTEST.COM
Sep 25 02:31:46 dist-test-slave-q8mg krb5kdc[16400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303506, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.60@KRBTEST.COM
I20260925 02:31:46.836613 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.822937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:39011 (local address 127.10.13.61:45569)
0925 02:31:46.822983 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.822985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.824138 (+ 1153us) server_negotiation.cc:407] Connection header received
0925 02:31:46.824257 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.824258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.824270 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.824316 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.826094 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.826806 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.827892 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.828225 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.828301 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.831524 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.831525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.831525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.831533 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.835227 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.835716 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.835717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.835717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.835770 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.835989 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.835990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.835991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.836116 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.836184 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.836424 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.836505 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:31:46.836671 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.822716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:40299 (local address 127.10.13.60:33515)
0925 02:31:46.822763 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.822765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.822892 (+ 127us) server_negotiation.cc:407] Connection header received
0925 02:31:46.824030 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.824031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.824043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.824098 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.824970 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.825534 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.827045 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.827362 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.827401 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.832516 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.832517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.832517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.832526 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.834453 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.835580 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.835581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.835582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.835639 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.836011 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.836012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.836012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.836210 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.836267 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.836440 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.836564 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:31:46.836997 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.822542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:41363 (local address 127.10.13.62:39793)
0925 02:31:46.822592 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.822594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.823086 (+ 492us) server_negotiation.cc:407] Connection header received
0925 02:31:46.824003 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.824004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.824019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.824079 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.824965 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.825746 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.827038 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.827338 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.827399 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.832086 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.832088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.832088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.832096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.834236 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.834783 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.834785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.834786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.834844 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.835132 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.835133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.835134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.835279 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.835340 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.836786 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.836875 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:46.837406 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.822570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:41363)
0925 02:31:46.822998 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.823007 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.823902 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.824152 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.824156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.824628 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.824826 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.824832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.825831 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.825832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.826939 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.826941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.827471 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.827472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.827616 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.828386 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.828390 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.831992 (+ 3602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.834332 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.834340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.834343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.834682 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.834916 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.834917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.834918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.835057 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.835366 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.835372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.835560 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.835695 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.837093 (+ 1398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":31,"spinlock_wait_cycles":7552,"thread_start_us":120,"threads_started":1}
I20260925 02:31:46.837852 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.822878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.10.13.3:39011)
0925 02:31:46.824077 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.824089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.824174 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.824331 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.824332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.824611 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.824825 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.824830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.826913 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.826914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.827814 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.827816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.828294 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.828295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.828366 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.828753 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.828758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.831404 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.835307 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.835309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.835309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.835500 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.835837 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.835838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.835838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.835916 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.836212 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.836213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.836256 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.836319 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.837735 (+ 1416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":1296,"thread_start_us":65,"threads_started":1}
I20260925 02:31:46.837898 17685 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39793
I20260925 02:31:46.837980 17685 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.838187 17685 heartbeater.cc:511] Master 127.10.13.62:39793 requested a full tablet report, sending...
I20260925 02:31:46.838369 17686 heartbeater.cc:347] Connected to a master server at 127.10.13.61:45569
I20260925 02:31:46.838430 17686 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.838500 17686 heartbeater.cc:511] Master 127.10.13.61:45569 requested a full tablet report, sending...
I20260925 02:31:46.838589 16445 ts_manager.cc:194] Re-registered known tserver with Master: b1da9111651a4f6085b2cccaff0a2325 (127.10.13.3:42285)
I20260925 02:31:46.838769 17046 ts_manager.cc:194] Registered new tserver with Master: b1da9111651a4f6085b2cccaff0a2325 (127.10.13.3:42285)
I20260925 02:31:46.838956 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.826304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.10.13.61:45569)
0925 02:31:46.827043 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.827045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.827051 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:31:46.827070 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.827071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.827082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.827134 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.829187 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.829935 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.830951 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.831239 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.831297 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.832857 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.832858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.832859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.832888 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.836198 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.837900 (+ 1702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.837901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.837901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.837972 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.838206 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.838206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.838207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.838338 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.838416 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.838747 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.838825 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":53,"threads_started":1}
I20260925 02:31:46.839092 16445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:41363
I20260925 02:31:46.839533 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.826645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.10.13.60:33515)
0925 02:31:46.826850 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.826853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.828056 (+ 1203us) server_negotiation.cc:407] Connection header received
0925 02:31:46.828080 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.828081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.828094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.828148 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.828954 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.830474 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.831523 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.831821 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.831861 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.833813 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.833814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.833814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.833822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.836816 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.837246 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.837247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.837248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.837308 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.837520 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.837521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.837521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.837656 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.837755 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.839356 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.839437 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":48,"threads_started":1}
I20260925 02:31:46.840442 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.828101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.10.13.62:39793)
0925 02:31:46.828698 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.828700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.829031 (+ 331us) server_negotiation.cc:407] Connection header received
0925 02:31:46.829054 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.829054 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.829066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.829122 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:46.829836 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.830741 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.835170 (+ 4429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.835532 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.835600 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.837548 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:46.837549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:46.837549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:46.837556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:46.839340 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.839756 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.839757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.839757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.839810 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:46.840015 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:46.840016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:46.840016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:46.840110 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:46.840186 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.840326 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.840384 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":88,"threads_started":1}
I20260925 02:31:46.840607 17688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.822571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.10.13.3:40299)
0925 02:31:46.822766 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.822817 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.823882 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.824153 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.824157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.824614 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:46.824826 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.824832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.825647 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.825648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.826939 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.826941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.827655 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.827656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.827713 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.828376 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:46.828386 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:46.832422 (+ 4036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:46.834822 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.834823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.834824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.835071 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.835887 (+ 816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:46.835888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:46.835888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:46.835932 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:46.836291 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:46.836292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:46.836329 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.836540 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.840464 (+ 3924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":3841,"thread_start_us":91,"threads_started":1}
I20260925 02:31:46.840973 17684 heartbeater.cc:347] Connected to a master server at 127.10.13.60:33515
I20260925 02:31:46.841027 17684 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:46.841099 17684 heartbeater.cc:511] Master 127.10.13.60:33515 requested a full tablet report, sending...
I20260925 02:31:46.841387 17232 ts_manager.cc:194] Registered new tserver with Master: b1da9111651a4f6085b2cccaff0a2325 (127.10.13.3:42285)
I20260925 02:31:46.842204 16445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:46.844408 16445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:46.845582 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.842658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.10.13.60:33515)
0925 02:31:46.842693 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.842695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.842703 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:46.842752 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.842753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.842810 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.842883 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:46.843037 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.843505 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.844695 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.845256 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.845348 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.845443 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.845488 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.845524 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:31:46.845585 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.842607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.0.0.1:39552)
0925 02:31:46.842653 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.842663 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.842696 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.842897 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.842897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.842899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:46.842936 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.842937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.843592 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.843593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.844597 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.844600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.845361 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.845362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.845423 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.845426 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.845480 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.845522 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:31:46.846141 17242 raft_consensus.cc:1203] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 02:31:46.846326 17271 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1356f4fefc949c1b76552c5092f0dd2. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.846423 17271 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: This master's current role is: LEARNER
I20260925 02:31:46.846755 17006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: NON_VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: true } } }
I20260925 02:31:46.846841 17006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:46.848264 17008 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: attempting to promote NON_VOTER adaf6e52e1f04faf97c0a89bd2975ffe to VOTER
I20260925 02:31:46.848381 17008 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } }
I20260925 02:31:46.848722 17056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Refusing update from remote peer e1356f4fefc949c1b76552c5092f0dd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260925 02:31:46.848913 17196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 02:31:46.849445 17241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 LEARNER]: Refusing update from remote peer e1356f4fefc949c1b76552c5092f0dd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260925 02:31:46.849649 17196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 02:31:46.849778 17196 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:46.850065 17056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:46.850109 17241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:46.850755 17196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:46.850826 17196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:46.850873 17196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:46.850908 17196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:46.851060 17195 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:46.851080 17271 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:46.851140 17195 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:31:46.851152 17271 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:31:46.854111 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.848852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:41771 (local address 127.0.0.1:34398)
0925 02:31:46.848998 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.849010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.849050 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.850097 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.850098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.850100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:46.850149 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.850150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.851829 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.851830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.852971 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.852974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.853879 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.853880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.853957 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.853961 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.854006 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.854044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260925 02:31:46.854210 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.849145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.10.13.2:41771)
0925 02:31:46.849915 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.849921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.849932 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:46.849976 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.849978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.849997 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.850219 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:46.850278 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.851462 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.853059 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.853788 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.853880 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.854048 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.854076 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.854118 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":74,"threads_started":1}
I20260925 02:31:46.854890 17482 tablet_service.cc:1467] Processing CreateTablet for tablet eef317bc359b417fa029264138ff9968 (DEFAULT_TABLE table=test-table [id=8876bc08c3ff4cc3b2cc80a6f8def3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:46.855330 17482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef317bc359b417fa029264138ff9968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:46.856592 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.849652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:42285 (local address 127.0.0.1:55246)
0925 02:31:46.851871 (+ 2219us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.851880 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.851927 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.852174 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.852175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.852178 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:46.852236 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.852238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.853664 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.853664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.855234 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.855238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.856325 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.856326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.856390 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.856395 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.856463 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.856502 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2204,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260925 02:31:46.856594 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.849723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.10.13.3:42285)
0925 02:31:46.851595 (+ 1872us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.851600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.851947 (+ 347us) server_negotiation.cc:407] Connection header received
0925 02:31:46.851995 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.851997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.852015 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.852105 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:46.852298 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.853573 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.855346 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.856233 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.856308 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.856384 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.856477 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.856516 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1832,"thread_start_us":100,"threads_started":1}
I20260925 02:31:46.857228 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.848852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:38135 (local address 127.0.0.1:53100)
0925 02:31:46.851662 (+ 2810us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.851673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.851717 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.852184 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.852185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.852187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:46.852277 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.852279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.853418 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.853419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.854614 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.854617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.856991 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.856993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.857066 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.857071 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.857115 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.857159 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2793,"mutex_wait_us":674,"thread_start_us":71,"threads_started":1}
I20260925 02:31:46.857489 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.849034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.10.13.1:38135)
0925 02:31:46.852036 (+ 3002us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.852040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.852047 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:31:46.852078 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.852080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.852092 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.852187 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:46.852345 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.853331 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.855795 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.856397 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.856463 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.856878 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.857359 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.857418 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2967,"thread_start_us":88,"threads_started":1}
I20260925 02:31:46.857496 17618 tablet_service.cc:1467] Processing CreateTablet for tablet eef317bc359b417fa029264138ff9968 (DEFAULT_TABLE table=test-table [id=8876bc08c3ff4cc3b2cc80a6f8def3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:46.857816 17618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef317bc359b417fa029264138ff9968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:46.858043 17345 tablet_service.cc:1467] Processing CreateTablet for tablet eef317bc359b417fa029264138ff9968 (DEFAULT_TABLE table=test-table [id=8876bc08c3ff4cc3b2cc80a6f8def3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:46.858284 17345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef317bc359b417fa029264138ff9968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:46.859097 17713 tablet_bootstrap.cc:492] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Bootstrap starting.
I20260925 02:31:46.859792 17713 tablet_bootstrap.cc:654] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:46.860260 17713 log.cc:824] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:46.860903 17714 tablet_bootstrap.cc:492] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: Bootstrap starting.
I20260925 02:31:46.861459 17713 tablet_bootstrap.cc:492] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: No bootstrap required, opened a new log
I20260925 02:31:46.861532 17713 ts_tablet_manager.cc:1397] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:31:46.861591 17714 tablet_bootstrap.cc:654] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:46.861951 17714 log.cc:824] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:46.862711 17714 tablet_bootstrap.cc:492] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: No bootstrap required, opened a new log
I20260925 02:31:46.862797 17714 ts_tablet_manager.cc:1397] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:46.863914 17713 raft_consensus.cc:348] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.864044 17713 raft_consensus.cc:374] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:46.864066 17713 raft_consensus.cc:729] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc1d98e39584e11beb4a6073d74f9a5, State: Initialized, Role: FOLLOWER
I20260925 02:31:46.864199 17713 consensus_queue.cc:260] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.864331 17714 raft_consensus.cc:348] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.864478 17713 ts_tablet_manager.cc:1428] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:46.864494 17714 raft_consensus.cc:374] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:46.864532 17714 raft_consensus.cc:729] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1da9111651a4f6085b2cccaff0a2325, State: Initialized, Role: FOLLOWER
I20260925 02:31:46.864542 17549 heartbeater.cc:503] Master 127.10.13.62:39793 was elected leader, sending a full tablet report...
I20260925 02:31:46.864651 17714 consensus_queue.cc:260] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.864913 17714 ts_tablet_manager.cc:1428] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:31:46.865192 17685 heartbeater.cc:503] Master 127.10.13.62:39793 was elected leader, sending a full tablet report...
I20260925 02:31:46.865402 17716 tablet_bootstrap.cc:492] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: Bootstrap starting.
I20260925 02:31:46.866119 17716 tablet_bootstrap.cc:654] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:46.866451 17716 log.cc:824] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:46.867218 17716 tablet_bootstrap.cc:492] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: No bootstrap required, opened a new log
I20260925 02:31:46.867268 17716 ts_tablet_manager.cc:1397] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:46.868800 17716 raft_consensus.cc:348] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.868992 17716 raft_consensus.cc:374] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:46.869059 17716 raft_consensus.cc:729] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b243f2481bfe49279f507a3f9421aa35, State: Initialized, Role: FOLLOWER
I20260925 02:31:46.869179 17716 consensus_queue.cc:260] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.869436 17716 ts_tablet_manager.cc:1428] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:31:46.869500 17412 heartbeater.cc:503] Master 127.10.13.62:39793 was elected leader, sending a full tablet report...
W20260925 02:31:46.935411 17551 tablet.cc:2369] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:46.973227 17719 raft_consensus.cc:482] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:46.973397 17719 raft_consensus.cc:504] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.973776 17719 leader_election.cc:288] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771), b243f2481bfe49279f507a3f9421aa35 (127.10.13.1:38135)
I20260925 02:31:46.976852 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.974128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:42743 (local address 127.10.13.2:41771)
0925 02:31:46.974184 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.974186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.974195 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:46.974228 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.974229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.974240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.974299 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:46.974459 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.975065 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.976094 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.976470 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.976557 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.976569 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.976731 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.976771 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:31:46.976912 17688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.973956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:41771 (local address 127.10.13.3:42743)
0925 02:31:46.974021 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.974033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.974088 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.974315 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.974316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.974317 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:46.974383 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.974384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.975173 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.975174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.976010 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.976013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.976556 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.976557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.976641 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.976644 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.976721 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.976762 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260925 02:31:46.976902 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.974076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:35563 (local address 127.10.13.1:38135)
0925 02:31:46.974126 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.974128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.974138 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:46.974169 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.974170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.974182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.974264 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:46.974450 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.975058 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.976119 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.976484 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.976584 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.976594 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.976770 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.976809 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:46.976924 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.974008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:38135 (local address 127.10.13.3:35563)
0925 02:31:46.974049 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:46.974058 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:46.974088 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:46.974285 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:46.974286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:46.974288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:46.974367 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.974369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.975145 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.975147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.976036 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:46.976039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.976589 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.976590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:46.976681 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.976684 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:46.976751 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:31:46.976793 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":25}
I20260925 02:31:46.977299 17502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef317bc359b417fa029264138ff9968" candidate_uuid: "b1da9111651a4f6085b2cccaff0a2325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc1d98e39584e11beb4a6073d74f9a5" is_pre_election: true
I20260925 02:31:46.977432 17502 raft_consensus.cc:2454] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1da9111651a4f6085b2cccaff0a2325 in term 0.
I20260925 02:31:46.977660 17365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef317bc359b417fa029264138ff9968" candidate_uuid: "b1da9111651a4f6085b2cccaff0a2325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b243f2481bfe49279f507a3f9421aa35" is_pre_election: true
I20260925 02:31:46.977730 17574 leader_election.cc:302] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1da9111651a4f6085b2cccaff0a2325, efc1d98e39584e11beb4a6073d74f9a5; no voters:
I20260925 02:31:46.977779 17365 raft_consensus.cc:2454] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1da9111651a4f6085b2cccaff0a2325 in term 0.
I20260925 02:31:46.977895 17719 raft_consensus.cc:2781] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:46.977952 17719 raft_consensus.cc:482] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:46.977980 17719 raft_consensus.cc:3037] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:46.978693 17719 raft_consensus.cc:504] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.978853 17719 leader_election.cc:288] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [CANDIDATE]: Term 1 election: Requested vote from peers efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771), b243f2481bfe49279f507a3f9421aa35 (127.10.13.1:38135)
I20260925 02:31:46.978994 17502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef317bc359b417fa029264138ff9968" candidate_uuid: "b1da9111651a4f6085b2cccaff0a2325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc1d98e39584e11beb4a6073d74f9a5"
I20260925 02:31:46.979025 17365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef317bc359b417fa029264138ff9968" candidate_uuid: "b1da9111651a4f6085b2cccaff0a2325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b243f2481bfe49279f507a3f9421aa35"
I20260925 02:31:46.979087 17365 raft_consensus.cc:3037] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:46.979085 17502 raft_consensus.cc:3037] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:46.980020 17502 raft_consensus.cc:2454] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1da9111651a4f6085b2cccaff0a2325 in term 1.
I20260925 02:31:46.980242 17574 leader_election.cc:302] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1da9111651a4f6085b2cccaff0a2325, efc1d98e39584e11beb4a6073d74f9a5; no voters:
I20260925 02:31:46.980337 17719 raft_consensus.cc:2781] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:46.980538 17719 raft_consensus.cc:686] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 1 LEADER]: Becoming Leader. State: Replica: b1da9111651a4f6085b2cccaff0a2325, State: Running, Role: LEADER
I20260925 02:31:46.980669 17365 raft_consensus.cc:2454] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1da9111651a4f6085b2cccaff0a2325 in term 1.
I20260925 02:31:46.980640 17719 consensus_queue.cc:237] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } }
I20260925 02:31:46.981839 16445 catalog_manager.cc:5495] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1da9111651a4f6085b2cccaff0a2325 (127.10.13.3). New cstate: current_term: 1 leader_uuid: "b1da9111651a4f6085b2cccaff0a2325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } health_report { overall_health: HEALTHY } } }
I20260925 02:31:46.982421 17196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1356f4fefc949c1b76552c5092f0dd2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:46.982532 17196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:46.998189 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:46.995891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.10.13.3:42285)
0925 02:31:46.995952 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:46.995954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:46.995979 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:31:46.996012 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:46.996013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:46.996031 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:46.996085 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:46.996201 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.996663 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.997431 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:46.997678 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:46.997732 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:46.997863 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:46.997962 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:46.998023 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:46.998059 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 02:31:46.998078 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:31:47.000041 17365 raft_consensus.cc:1261] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 1 FOLLOWER]: Refusing update from remote peer b1da9111651a4f6085b2cccaff0a2325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:47.000041 17502 raft_consensus.cc:1261] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 1 FOLLOWER]: Refusing update from remote peer b1da9111651a4f6085b2cccaff0a2325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:47.000416 17719 consensus_queue.cc:1035] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:47.000568 17722 consensus_queue.cc:1035] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:47.005079 17727 mvcc.cc:205] Tried to move back new op lower bound from 7333083164669456384 to 7333083164593364992. Current Snapshot: MvccSnapshot[applied={T|T < 7333083164669456384 or (T in {7333083164669456384})}]
I20260925 02:31:47.008775 17726 mvcc.cc:205] Tried to move back new op lower bound from 7333083164669456384 to 7333083164593364992. Current Snapshot: MvccSnapshot[applied={T|T < 7333083164669456384}]
I20260925 02:31:47.010046 16444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:47.010169 16444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:47.012166 16444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1356f4fefc949c1b76552c5092f0dd2: Sending DeleteTablet for 3 replicas of tablet eef317bc359b417fa029264138ff9968
I20260925 02:31:47.012979 17482 tablet_service.cc:1514] Processing DeleteTablet for tablet eef317bc359b417fa029264138ff9968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:47 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:34398
I20260925 02:31:47.013010 17345 tablet_service.cc:1514] Processing DeleteTablet for tablet eef317bc359b417fa029264138ff9968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:47 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:53100
I20260925 02:31:47.013355 17618 tablet_service.cc:1514] Processing DeleteTablet for tablet eef317bc359b417fa029264138ff9968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:47 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:55246
I20260925 02:31:47.013480 17617 tablet_service.cc:1514] Processing DeleteTablet for tablet eef317bc359b417fa029264138ff9968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:47 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:55246
I20260925 02:31:47.013810 17734 tablet_replica.cc:330] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: stopping tablet replica
I20260925 02:31:47.013969 17734 raft_consensus.cc:2229] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:47.013989 17735 tablet_replica.cc:330] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: stopping tablet replica
I20260925 02:31:47.014073 17735 raft_consensus.cc:2229] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:47.014088 17734 raft_consensus.cc:2258] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:47.014264 17735 raft_consensus.cc:2258] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:47.014791 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 16413
I20260925 02:31:47.015720 17736 tablet_replica.cc:330] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: stopping tablet replica
I20260925 02:31:47.015833 17736 raft_consensus.cc:2229] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:47.015962 17736 raft_consensus.cc:2258] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:47.016356 17735 ts_tablet_manager.cc:1905] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:47.016461 17736 ts_tablet_manager.cc:1905] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:47.018715 17736 ts_tablet_manager.cc:1918] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:47.018783 17736 log.cc:1196] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/eef317bc359b417fa029264138ff9968
I20260925 02:31:47.019038 17736 ts_tablet_manager.cc:1939] T eef317bc359b417fa029264138ff9968 P b1da9111651a4f6085b2cccaff0a2325: Deleting consensus metadata
I20260925 02:31:47.020048 17735 ts_tablet_manager.cc:1918] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:47.020130 17735 log.cc:1196] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/eef317bc359b417fa029264138ff9968
I20260925 02:31:47.020421 17735 ts_tablet_manager.cc:1939] T eef317bc359b417fa029264138ff9968 P b243f2481bfe49279f507a3f9421aa35: Deleting consensus metadata
W20260925 02:31:47.021603 17573 connection.cc:535] server connection from 127.0.0.1:55246 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 02:31:47.021790 17298 connection.cc:535] server connection from 127.0.0.1:53100 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:31:47.023077 17734 ts_tablet_manager.cc:1905] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:47.024835 17734 ts_tablet_manager.cc:1918] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:47.024942 17734 log.cc:1196] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/eef317bc359b417fa029264138ff9968
I20260925 02:31:47.025206 17734 ts_tablet_manager.cc:1939] T eef317bc359b417fa029264138ff9968 P efc1d98e39584e11beb4a6073d74f9a5: Deleting consensus metadata
W20260925 02:31:47.025697 17435 connection.cc:441] server connection from 127.0.0.1:34398 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34398 (request call id 1) could send its response
I20260925 02:31:47.026954 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:47.014486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.10.13.60:33515)
0925 02:31:47.014554 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:47.014556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:47.014564 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:47.014586 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:47.014587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:47.014597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:47.014642 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:47.016704 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:47.017479 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:47.018542 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:47.018850 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:47.018895 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:47.020766 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:47.020767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:47.020768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:47.020775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:47.022738 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:47.026047 (+ 3309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:47.026048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:47.026049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:47.026113 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:47.026315 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:47.026316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:47.026316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:47.026443 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:47.026522 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:47.026747 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 02:31:47.026846 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:31:47.031672 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:47.023681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.10.13.60:33515)
0925 02:31:47.023719 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:47.023721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:47.023994 (+ 273us) server_negotiation.cc:407] Connection header received
0925 02:31:47.024091 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:47.024092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:47.024104 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:47.024151 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:47.024787 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:47.025593 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:47.026883 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:47.027176 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:47.027213 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:47.028996 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:47.028997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:47.028998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:47.029004 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:47.030506 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:47.030907 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:47.030908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:47.030909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:47.030961 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:47.031164 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:47.031165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:47.031166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:47.031292 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:47.031396 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:47.031502 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 02:31:47.031576 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:31:47.031836 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:47.023429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.10.13.61:45569)
0925 02:31:47.023474 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:47.023476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:47.023589 (+ 113us) server_negotiation.cc:407] Connection header received
0925 02:31:47.023705 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:47.023706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:47.023721 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:47.023751 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:47.024845 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:47.025659 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:47.026757 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:47.027067 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:47.027106 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:47.028704 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:47.028706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:47.028706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:47.028714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:47.030521 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:47.030980 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:47.030981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:47.030981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:47.031039 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:47.031214 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:47.031215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:47.031215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:47.031354 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:47.031447 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:47.031650 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 02:31:47.031739 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:48.009426 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.009177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.2:59141)
0925 02:31:48.009307 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.009382 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260925 02:31:48.009426 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.009191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:54183)
0925 02:31:48.009310 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.009384 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
W20260925 02:31:48.009698 17549 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:39793 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
W20260925 02:31:48.009791 17412 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:39793 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:48.013229 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.012990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:45361)
0925 02:31:48.013119 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.013191 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":51,"threads_started":1}
W20260925 02:31:48.013458 17685 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:39793 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:48.512415 17748 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1356f4fefc949c1b76552c5092f0dd2)
I20260925 02:31:48.512518 17748 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } }
I20260925 02:31:48.512820 17748 leader_election.cc:288] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793), adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60:33515)
I20260925 02:31:48.513011 17749 raft_consensus.cc:482] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1356f4fefc949c1b76552c5092f0dd2)
I20260925 02:31:48.513098 17749 raft_consensus.cc:504] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } }
I20260925 02:31:48.513294 17749 leader_election.cc:288] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793), ea1da0ece3a1414aa0bb4138754dea89 (127.10.13.61:45569)
I20260925 02:31:48.513463 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.512902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37352)
0925 02:31:48.513167 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.513237 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":63,"threads_started":1}
W20260925 02:31:48.513609 17033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:48.513988 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.513356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37362)
0925 02:31:48.513504 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.513944 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":43,"threads_started":1}
I20260925 02:31:48.514355 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.514240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37380)
0925 02:31:48.514297 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.514329 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:31:48.514454 17033 leader_election.cc:334] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793): Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
W20260925 02:31:48.514391 17219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:48.515021 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.514899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37396)
0925 02:31:48.514958 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.514991 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:31:48.516096 17219 leader_election.cc:334] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793): Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:48.516912 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.513506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.10.13.61:45569)
0925 02:31:48.513853 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:48.513855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:48.513864 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:48.513886 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:48.513887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:48.513898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:48.513992 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:48.514192 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:48.514913 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:48.516104 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:48.516550 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:48.516602 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:48.516612 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:48.516788 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:31:48.516832 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":45,"threads_started":1}
I20260925 02:31:48.516925 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.513409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:45569 (local address 127.0.0.1:51318)
0925 02:31:48.513588 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.513598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:48.513654 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:48.514047 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:48.514048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:48.514050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:48.514111 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:48.514113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:48.515015 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:48.515016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:48.515993 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:48.515996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:48.516627 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:48.516628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:48.516686 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:48.516691 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:48.516756 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:31:48.516801 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":48,"threads_started":1}
I20260925 02:31:48.517066 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.513043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:33515 (local address 127.0.0.1:39582)
0925 02:31:48.513255 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.513268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:48.513379 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:48.514094 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:48.514095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:48.514097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:48.514155 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:48.514157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:48.515317 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:48.515318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:48.516199 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:48.516201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:48.516808 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:48.516809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:48.516862 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:48.516865 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:48.516949 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:31:48.516995 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":57,"threads_started":1}
I20260925 02:31:48.517261 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.513188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.10.13.60:33515)
0925 02:31:48.513279 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:48.513282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:48.513983 (+ 701us) server_negotiation.cc:407] Connection header received
0925 02:31:48.514005 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:48.514005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:48.514016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:48.514068 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:48.514317 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:48.515212 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:48.516287 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:48.516735 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:48.516796 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:48.516804 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:48.517092 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 02:31:48.517214 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":42,"threads_started":1}
I20260925 02:31:48.517326 17056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea1da0ece3a1414aa0bb4138754dea89" is_pre_election: true
I20260925 02:31:48.517423 17056 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adaf6e52e1f04faf97c0a89bd2975ffe in term 1.
I20260925 02:31:48.517469 17241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea1da0ece3a1414aa0bb4138754dea89" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" is_pre_election: true
I20260925 02:31:48.517576 17216 leader_election.cc:302] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: adaf6e52e1f04faf97c0a89bd2975ffe, ea1da0ece3a1414aa0bb4138754dea89; no voters: e1356f4fefc949c1b76552c5092f0dd2
I20260925 02:31:48.517591 17241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea1da0ece3a1414aa0bb4138754dea89 in term 1.
I20260925 02:31:48.517722 17749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:31:48.517765 17749 raft_consensus.cc:482] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1356f4fefc949c1b76552c5092f0dd2)
I20260925 02:31:48.517788 17749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:31:48.517791 17030 leader_election.cc:302] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: adaf6e52e1f04faf97c0a89bd2975ffe, ea1da0ece3a1414aa0bb4138754dea89; no voters: e1356f4fefc949c1b76552c5092f0dd2
I20260925 02:31:48.517895 17748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:31:48.517956 17748 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1356f4fefc949c1b76552c5092f0dd2)
I20260925 02:31:48.517980 17748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:31:48.518543 17749 raft_consensus.cc:504] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } }
I20260925 02:31:48.518683 17748 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } }
I20260925 02:31:48.518813 17748 leader_election.cc:288] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 2 election: Requested vote from peers e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793), adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60:33515)
I20260925 02:31:48.518816 17056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea1da0ece3a1414aa0bb4138754dea89"
I20260925 02:31:48.518801 17749 leader_election.cc:288] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [CANDIDATE]: Term 2 election: Requested vote from peers e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793), ea1da0ece3a1414aa0bb4138754dea89 (127.10.13.61:45569)
I20260925 02:31:48.518931 17056 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate adaf6e52e1f04faf97c0a89bd2975ffe in current term 2: Already voted for candidate ea1da0ece3a1414aa0bb4138754dea89 in this term.
I20260925 02:31:48.518929 17241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea1da0ece3a1414aa0bb4138754dea89" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe"
I20260925 02:31:48.519019 17241 raft_consensus.cc:2379] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ea1da0ece3a1414aa0bb4138754dea89 in current term 2: Already voted for candidate adaf6e52e1f04faf97c0a89bd2975ffe in this term.
I20260925 02:31:48.519042 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.518955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37414)
0925 02:31:48.518993 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.519022 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 02:31:48.519253 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.518746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37402)
0925 02:31:48.518828 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.519223 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:31:48.519330 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.519248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37430)
0925 02:31:48.519283 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.519307 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 02:31:48.519394 17033 leader_election.cc:334] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793): Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:48.519436 17033 leader_election.cc:302] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea1da0ece3a1414aa0bb4138754dea89; no voters: adaf6e52e1f04faf97c0a89bd2975ffe, e1356f4fefc949c1b76552c5092f0dd2
I20260925 02:31:48.519519 17748 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:31:48.519659 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:48.519488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37446)
0925 02:31:48.519611 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:48.519639 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260925 02:31:48.519748 17219 leader_election.cc:334] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793): Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:48.519814 17219 leader_election.cc:302] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: adaf6e52e1f04faf97c0a89bd2975ffe; no voters: e1356f4fefc949c1b76552c5092f0dd2, ea1da0ece3a1414aa0bb4138754dea89
I20260925 02:31:48.519897 17749 raft_consensus.cc:2726] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:31:49.010331 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:49.010052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.2:57677)
0925 02:31:49.010220 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:49.010270 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260925 02:31:49.010537 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:49.010288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:55937)
0925 02:31:49.010438 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:49.010485 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260925 02:31:49.013993 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:49.013767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:35549)
0925 02:31:49.013904 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:49.013952 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260925 02:31:50.011210 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.010948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.2:58637)
0925 02:31:50.011089 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.011134 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260925 02:31:50.011308 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.011073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:47025)
0925 02:31:50.011222 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.011258 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260925 02:31:50.014712 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.014484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:56889)
0925 02:31:50.014618 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.014663 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260925 02:31:50.542382 17764 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:50.542508 17764 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } }
I20260925 02:31:50.542842 17764 leader_election.cc:288] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793), adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60:33515)
I20260925 02:31:50.543078 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.542865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37692)
0925 02:31:50.542969 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.543034 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260925 02:31:50.543035 17241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea1da0ece3a1414aa0bb4138754dea89" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" is_pre_election: true
I20260925 02:31:50.543144 17241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea1da0ece3a1414aa0bb4138754dea89 in term 2.
I20260925 02:31:50.543314 17030 leader_election.cc:302] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adaf6e52e1f04faf97c0a89bd2975ffe, ea1da0ece3a1414aa0bb4138754dea89; no voters:
I20260925 02:31:50.543407 17764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 02:31:50.543452 17764 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:50.543467 17764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:31:50.543661 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.543567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37694)
0925 02:31:50.543604 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.543636 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 02:31:50.543749 17033 leader_election.cc:334] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793): Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:50.544344 17764 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } }
I20260925 02:31:50.544481 17764 leader_election.cc:288] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 3 election: Requested vote from peers e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793), adaf6e52e1f04faf97c0a89bd2975ffe (127.10.13.60:33515)
I20260925 02:31:50.544652 17241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea1da0ece3a1414aa0bb4138754dea89" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe"
I20260925 02:31:50.544730 17241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:31:50.544992 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.544599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37704)
0925 02:31:50.544940 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.544970 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322}
I20260925 02:31:50.545369 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.545257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37712)
0925 02:31:50.545310 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.545341 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:31:50.545485 17033 leader_election.cc:334] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793): Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
I20260925 02:31:50.545656 17241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea1da0ece3a1414aa0bb4138754dea89 in term 3.
I20260925 02:31:50.545819 17030 leader_election.cc:302] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: adaf6e52e1f04faf97c0a89bd2975ffe, ea1da0ece3a1414aa0bb4138754dea89; no voters: e1356f4fefc949c1b76552c5092f0dd2
I20260925 02:31:50.545933 17764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:31:50.546048 17764 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [term 3 LEADER]: Becoming Leader. State: Replica: ea1da0ece3a1414aa0bb4138754dea89, State: Running, Role: LEADER
I20260925 02:31:50.546133 17764 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } }
I20260925 02:31:50.546494 17767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea1da0ece3a1414aa0bb4138754dea89. Latest consensus state: current_term: 3 leader_uuid: "ea1da0ece3a1414aa0bb4138754dea89" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:50.546615 17767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:50.546818 17769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:50.547228 17769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:50.547303 17769 catalog_manager.cc:1259] Loaded cluster ID: a42275f823954f4c9d9acda7185bab48
I20260925 02:31:50.547338 17769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:50.547535 17769 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:50.547622 17769 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Loaded TSK: 0
I20260925 02:31:50.547772 17769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:31:50.608858 17045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:50.610580 17045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:50.611722 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.611589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37724)
0925 02:31:50.611661 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.611694 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:31:50.611802 17241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [term 3 FOLLOWER]: Refusing update from remote peer ea1da0ece3a1414aa0bb4138754dea89: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 02:31:50.612035 17764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 02:31:50.612164 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.612037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.0.0.1:37734)
0925 02:31:50.612087 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.612119 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:31:50.612303 17033 consensus_peers.cc:487] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 -> Peer e1356f4fefc949c1b76552c5092f0dd2 (127.10.13.62:39793): Couldn't send request to peer e1356f4fefc949c1b76552c5092f0dd2. Status: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:31:50.612717 17771 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea1da0ece3a1414aa0bb4138754dea89. Latest consensus state: current_term: 3 leader_uuid: "ea1da0ece3a1414aa0bb4138754dea89" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:50.612838 17771 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:31:50.613217 17770 mvcc.cc:205] Tried to move back new op lower bound from 7333083179463331840 to 7333083179197526016. Current Snapshot: MvccSnapshot[applied={T|T < 7333083179463331840}]
I20260925 02:31:50.613363 17767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ea1da0ece3a1414aa0bb4138754dea89" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:50.613471 17767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:50.613648 17771 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ea1da0ece3a1414aa0bb4138754dea89" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e1356f4fefc949c1b76552c5092f0dd2" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39793 } } peers { permanent_uuid: "ea1da0ece3a1414aa0bb4138754dea89" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 45569 } attrs { promote: false } } peers { permanent_uuid: "adaf6e52e1f04faf97c0a89bd2975ffe" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 33515 } attrs { promote: false } } }
I20260925 02:31:50.613775 17771 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaf6e52e1f04faf97c0a89bd2975ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:31:50.618667 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.615394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:38135 (local address 127.0.0.1:53104)
0925 02:31:50.615540 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.615551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:50.615601 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:50.615775 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:50.615776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:50.615779 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:50.615843 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.615844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.616913 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.616914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.617772 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.617775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.618429 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.618430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.618501 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.618505 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:50.618557 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:31:50.618598 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":22}
I20260925 02:31:50.618906 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.615481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.10.13.1:38135)
0925 02:31:50.615597 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:50.615600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:50.615611 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:50.615667 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:50.615667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:50.615679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:50.615912 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:50.615955 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.616790 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.617875 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.618308 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.618429 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.618446 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:50.618788 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 02:31:50.618840 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260925 02:31:50.619232 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.615410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:41771 (local address 127.0.0.1:34412)
0925 02:31:50.615597 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.615606 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:50.615644 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:50.616119 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:50.616119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:50.616121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:50.616164 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.616165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.616819 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.616820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.618120 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.618123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.618973 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.618974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.619060 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.619065 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:50.619121 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:31:50.619163 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":71,"thread_start_us":78,"threads_started":1}
I20260925 02:31:50.619231 17345 tablet_service.cc:1467] Processing CreateTablet for tablet 75245bd1141e4b7096fb99ad45fb4326 (DEFAULT_TABLE table=test-table [id=68032c1f552243c2897a54dce8dc34df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:50.619275 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.615503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.10.13.2:41771)
0925 02:31:50.615739 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:50.615742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:50.615751 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:31:50.615780 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:50.615780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:50.615791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:50.616220 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:50.616255 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.616728 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.618323 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.618881 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.618971 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.618984 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:50.619141 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:31:50.619198 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":55,"threads_started":1}
I20260925 02:31:50.619374 17345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75245bd1141e4b7096fb99ad45fb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:50.619493 17482 tablet_service.cc:1467] Processing CreateTablet for tablet 75245bd1141e4b7096fb99ad45fb4326 (DEFAULT_TABLE table=test-table [id=68032c1f552243c2897a54dce8dc34df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:50.619596 17482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75245bd1141e4b7096fb99ad45fb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:50.619985 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.615486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:42285 (local address 127.0.0.1:55262)
0925 02:31:50.615950 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.615958 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:50.616024 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:50.616198 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:50.616199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:50.616201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:50.616254 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.616255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.617117 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.617118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.618216 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.618231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.619739 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.619740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.619810 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.619813 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:50.619879 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 02:31:50.619918 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":80,"threads_started":1}
I20260925 02:31:50.619990 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.615665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.10.13.3:42285)
0925 02:31:50.615865 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:50.615867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:50.616017 (+ 150us) server_negotiation.cc:407] Connection header received
0925 02:31:50.616078 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:50.616079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:50.616089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:50.616150 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:50.616322 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.617028 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.619021 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.619627 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.619691 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.619705 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:50.619886 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:31:50.619926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":57,"threads_started":1}
I20260925 02:31:50.620458 17617 tablet_service.cc:1467] Processing CreateTablet for tablet 75245bd1141e4b7096fb99ad45fb4326 (DEFAULT_TABLE table=test-table [id=68032c1f552243c2897a54dce8dc34df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:50.620602 17617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75245bd1141e4b7096fb99ad45fb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:50.620965 17782 tablet_bootstrap.cc:492] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35: Bootstrap starting.
I20260925 02:31:50.621376 17782 tablet_bootstrap.cc:654] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:50.621704 17783 tablet_bootstrap.cc:492] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: Bootstrap starting.
I20260925 02:31:50.621984 17782 tablet_bootstrap.cc:492] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35: No bootstrap required, opened a new log
I20260925 02:31:50.622040 17782 ts_tablet_manager.cc:1397] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:50.622203 17782 raft_consensus.cc:348] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.622292 17782 raft_consensus.cc:374] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:50.622316 17782 raft_consensus.cc:729] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b243f2481bfe49279f507a3f9421aa35, State: Initialized, Role: FOLLOWER
I20260925 02:31:50.622378 17785 tablet_bootstrap.cc:492] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: Bootstrap starting.
I20260925 02:31:50.622484 17782 consensus_queue.cc:260] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.622704 17782 ts_tablet_manager.cc:1428] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:50.622812 17785 tablet_bootstrap.cc:654] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:50.623155 17783 tablet_bootstrap.cc:654] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:50.623229 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.622953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:50133)
0925 02:31:50.623139 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.623179 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":142,"threads_started":1}
I20260925 02:31:50.623392 17785 tablet_bootstrap.cc:492] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: No bootstrap required, opened a new log
I20260925 02:31:50.623479 17785 ts_tablet_manager.cc:1397] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:50.623620 17786 raft_consensus.cc:482] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:50.623636 17785 raft_consensus.cc:348] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.623749 17785 raft_consensus.cc:374] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:50.623778 17785 raft_consensus.cc:729] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1da9111651a4f6085b2cccaff0a2325, State: Initialized, Role: FOLLOWER
I20260925 02:31:50.623801 17413 heartbeater.cc:503] Master 127.10.13.61:45569 was elected leader, sending a full tablet report...
I20260925 02:31:50.623817 17785 consensus_queue.cc:260] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.623936 17785 ts_tablet_manager.cc:1428] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:50.623710 17786 raft_consensus.cc:504] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.624604 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.624061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:46517)
0925 02:31:50.624422 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.624455 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344}
I20260925 02:31:50.625250 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.624209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:45047)
0925 02:31:50.625172 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.625215 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":72,"threads_started":1}
I20260925 02:31:50.625273 17783 tablet_bootstrap.cc:492] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: No bootstrap required, opened a new log
I20260925 02:31:50.625380 17783 ts_tablet_manager.cc:1397] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 02:31:50.625551 17686 heartbeater.cc:503] Master 127.10.13.61:45569 was elected leader, sending a full tablet report...
I20260925 02:31:50.625629 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.625538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:53503)
0925 02:31:50.625572 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.625608 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 02:31:50.625658 17783 raft_consensus.cc:348] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.625865 17783 raft_consensus.cc:374] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:50.625902 17783 raft_consensus.cc:729] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc1d98e39584e11beb4a6073d74f9a5, State: Initialized, Role: FOLLOWER
I20260925 02:31:50.625957 17783 consensus_queue.cc:260] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.626215 17783 ts_tablet_manager.cc:1428] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:31:50.625113 17786 leader_election.cc:288] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1da9111651a4f6085b2cccaff0a2325 (127.10.13.3:42285), efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771)
I20260925 02:31:50.626715 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.626503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.2:37471)
0925 02:31:50.626634 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.626668 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":85,"threads_started":1}
I20260925 02:31:50.626945 17550 heartbeater.cc:503] Master 127.10.13.61:45569 was elected leader, sending a full tablet report...
I20260925 02:31:50.629345 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.625111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:42285 (local address 127.10.13.1:57653)
0925 02:31:50.626084 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.626094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:50.626175 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:50.626307 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:50.626308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:50.626310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:50.626350 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.626352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.627205 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.627205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.628503 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.628506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.629080 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.629081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.629157 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.629161 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:50.629239 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:31:50.629279 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":95,"threads_started":1}
I20260925 02:31:50.629338 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.625350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:57653 (local address 127.10.13.3:42285)
0925 02:31:50.625525 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:50.625527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:50.626164 (+ 637us) server_negotiation.cc:407] Connection header received
0925 02:31:50.626230 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:50.626231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:50.626243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:50.626302 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:50.626440 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.627101 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.628585 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.628993 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.629079 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.629090 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:50.629248 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:31:50.629283 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260925 02:31:50.629746 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.625308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:42355 (local address 127.10.13.2:41771)
0925 02:31:50.625388 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:50.625390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:50.625397 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:31:50.625419 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:50.625420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:50.625430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:50.625478 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:50.627054 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.627879 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.629002 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.629357 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.629442 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.629456 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:50.629673 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 02:31:50.629707 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:31:50.629678 17638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75245bd1141e4b7096fb99ad45fb4326" candidate_uuid: "b243f2481bfe49279f507a3f9421aa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1da9111651a4f6085b2cccaff0a2325" is_pre_election: true
I20260925 02:31:50.629784 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.625077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:41771 (local address 127.10.13.1:42355)
0925 02:31:50.625246 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.625257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:50.625310 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:50.625983 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:50.625984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:50.625986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:50.626826 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.626829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.627980 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.627981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.628920 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:50.628922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.629518 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.629520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:50.629610 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.629615 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:50.629669 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:31:50.629713 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260925 02:31:50.629808 17638 raft_consensus.cc:2454] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b243f2481bfe49279f507a3f9421aa35 in term 0.
I20260925 02:31:50.629985 17299 leader_election.cc:302] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1da9111651a4f6085b2cccaff0a2325, b243f2481bfe49279f507a3f9421aa35; no voters:
I20260925 02:31:50.630035 17502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75245bd1141e4b7096fb99ad45fb4326" candidate_uuid: "b243f2481bfe49279f507a3f9421aa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc1d98e39584e11beb4a6073d74f9a5" is_pre_election: true
I20260925 02:31:50.630107 17502 raft_consensus.cc:2454] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b243f2481bfe49279f507a3f9421aa35 in term 0.
I20260925 02:31:50.630128 17786 raft_consensus.cc:2781] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:50.630177 17786 raft_consensus.cc:482] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:50.630198 17786 raft_consensus.cc:3037] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:50.630846 17786 raft_consensus.cc:504] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.630965 17786 leader_election.cc:288] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [CANDIDATE]: Term 1 election: Requested vote from peers b1da9111651a4f6085b2cccaff0a2325 (127.10.13.3:42285), efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771)
I20260925 02:31:50.631122 17502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75245bd1141e4b7096fb99ad45fb4326" candidate_uuid: "b243f2481bfe49279f507a3f9421aa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc1d98e39584e11beb4a6073d74f9a5"
I20260925 02:31:50.631148 17638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75245bd1141e4b7096fb99ad45fb4326" candidate_uuid: "b243f2481bfe49279f507a3f9421aa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1da9111651a4f6085b2cccaff0a2325"
I20260925 02:31:50.631198 17502 raft_consensus.cc:3037] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:50.631203 17638 raft_consensus.cc:3037] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:50.631949 17502 raft_consensus.cc:2454] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b243f2481bfe49279f507a3f9421aa35 in term 1.
I20260925 02:31:50.631950 17638 raft_consensus.cc:2454] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b243f2481bfe49279f507a3f9421aa35 in term 1.
I20260925 02:31:50.632122 17301 leader_election.cc:302] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b243f2481bfe49279f507a3f9421aa35, efc1d98e39584e11beb4a6073d74f9a5; no voters:
I20260925 02:31:50.632218 17786 raft_consensus.cc:2781] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:50.632268 17786 raft_consensus.cc:686] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [term 1 LEADER]: Becoming Leader. State: Replica: b243f2481bfe49279f507a3f9421aa35, State: Running, Role: LEADER
I20260925 02:31:50.632328 17786 consensus_queue.cc:237] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } }
I20260925 02:31:50.632563 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.632470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:40219)
0925 02:31:50.632511 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.632535 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 02:31:50.633061 17045 catalog_manager.cc:5495] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b243f2481bfe49279f507a3f9421aa35 (127.10.13.1). New cstate: current_term: 1 leader_uuid: "b243f2481bfe49279f507a3f9421aa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b243f2481bfe49279f507a3f9421aa35" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 } health_report { overall_health: UNKNOWN } } }
I20260925 02:31:50.641351 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.638788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.10.13.1:38135)
0925 02:31:50.638856 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:50.638858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:50.638890 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:31:50.638964 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:50.638970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:50.638988 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:50.639047 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:50.639198 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.639766 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.640522 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:50.640702 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:50.640757 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:50.640977 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:50.641111 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:50.641171 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:50.641246 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:31:50.641265 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:31:50.642455 17502 raft_consensus.cc:1261] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 1 FOLLOWER]: Refusing update from remote peer b243f2481bfe49279f507a3f9421aa35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:50.642613 17638 raft_consensus.cc:1261] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 1 FOLLOWER]: Refusing update from remote peer b243f2481bfe49279f507a3f9421aa35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:50.642720 17786 consensus_queue.cc:1035] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efc1d98e39584e11beb4a6073d74f9a5" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 41771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:50.642901 17790 consensus_queue.cc:1035] T 75245bd1141e4b7096fb99ad45fb4326 P b243f2481bfe49279f507a3f9421aa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1da9111651a4f6085b2cccaff0a2325" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 42285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:50.643468 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.643305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.2:48037)
0925 02:31:50.643392 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.643423 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 02:31:50.643947 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.643851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:37487)
0925 02:31:50.643893 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.643918 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 02:31:50.645406 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.645072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.1:60243)
0925 02:31:50.645343 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.645376 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260925 02:31:50.645927 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.645818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.2:52869)
0925 02:31:50.645867 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.645894 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:31:50.647013 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.646885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39793 (local address 127.10.13.3:58081)
0925 02:31:50.646942 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.646974 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:31:50.647154 17045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:50.647234 17045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:50.648342 17045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea1da0ece3a1414aa0bb4138754dea89: Sending DeleteTablet for 3 replicas of tablet 75245bd1141e4b7096fb99ad45fb4326
I20260925 02:31:50.648640 17482 tablet_service.cc:1514] Processing DeleteTablet for tablet 75245bd1141e4b7096fb99ad45fb4326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:50 UTC) from {username='oryx', principal='oryx/127.10.13.61@KRBTEST.COM'} at 127.0.0.1:34412
I20260925 02:31:50.648643 17345 tablet_service.cc:1514] Processing DeleteTablet for tablet 75245bd1141e4b7096fb99ad45fb4326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:50 UTC) from {username='oryx', principal='oryx/127.10.13.61@KRBTEST.COM'} at 127.0.0.1:53104
I20260925 02:31:50.648890 17617 tablet_service.cc:1514] Processing DeleteTablet for tablet 75245bd1141e4b7096fb99ad45fb4326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:50 UTC) from {username='oryx', principal='oryx/127.10.13.61@KRBTEST.COM'} at 127.0.0.1:55262
I20260925 02:31:50.649070 17810 tablet_replica.cc:330] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: stopping tablet replica
I20260925 02:31:50.649144 17810 raft_consensus.cc:2229] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:50.649194 17810 raft_consensus.cc:2258] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:50.650238 17810 ts_tablet_manager.cc:1905] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:50.650271 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 17193
I20260925 02:31:50.652227 17809 tablet_replica.cc:330] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: stopping tablet replica
I20260925 02:31:50.652292 17809 raft_consensus.cc:2229] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:50.652318 17810 ts_tablet_manager.cc:1918] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:50.652343 17809 raft_consensus.cc:2258] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:50.652359 17810 log.cc:1196] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/75245bd1141e4b7096fb99ad45fb4326
I20260925 02:31:50.652598 17810 ts_tablet_manager.cc:1939] T 75245bd1141e4b7096fb99ad45fb4326 P b1da9111651a4f6085b2cccaff0a2325: Deleting consensus metadata
I20260925 02:31:50.652702 17809 ts_tablet_manager.cc:1905] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:50.652904 17031 catalog_manager.cc:4841] TS b1da9111651a4f6085b2cccaff0a2325 (127.10.13.3:42285): tablet 75245bd1141e4b7096fb99ad45fb4326 (table test-table [id=68032c1f552243c2897a54dce8dc34df]) successfully deleted
I20260925 02:31:50.654557 17809 ts_tablet_manager.cc:1918] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:50.654642 17809 log.cc:1196] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/75245bd1141e4b7096fb99ad45fb4326
I20260925 02:31:50.654892 17809 ts_tablet_manager.cc:1939] T 75245bd1141e4b7096fb99ad45fb4326 P efc1d98e39584e11beb4a6073d74f9a5: Deleting consensus metadata
I20260925 02:31:50.655396 17033 catalog_manager.cc:4841] TS efc1d98e39584e11beb4a6073d74f9a5 (127.10.13.2:41771): tablet 75245bd1141e4b7096fb99ad45fb4326 (table test-table [id=68032c1f552243c2897a54dce8dc34df]) successfully deleted
I20260925 02:31:50.658486 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:50.658353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:38135 (local address 127.0.0.1:53126)
0925 02:31:50.658415 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:50.658446 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:38135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 02:31:50.658638 17031 catalog_manager.cc:4584] TS b243f2481bfe49279f507a3f9421aa35 (127.10.13.1:38135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75245bd1141e4b7096fb99ad45fb4326: Network error: Client connection negotiation failed: client connection to 127.10.13.1:38135: connect: Connection refused (error 111)
I20260925 02:31:50.658751 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 17419
I20260925 02:31:50.663838 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 17555
I20260925 02:31:50.670305 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 16912
I20260925 02:31:50.675876 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 17100
2026-09-25T02:31:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6689 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:50 dist-test-slave-q8mg krb5kdc[17813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:50 dist-test-slave-q8mg krb5kdc[17813](info): set up 2 sockets
Sep 25 02:31:50 dist-test-slave-q8mg krb5kdc[17813](info): commencing operation
krb5kdc: starting...
W20260925 02:31:51.739962 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:51 dist-test-slave-q8mg krb5kdc[17813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.62.keytab.
Entry for principal oryx/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.62.keytab.
I20260925 02:31:51.860502 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39055
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.62.keytab
--principal=oryx/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:51.944306 17826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:51.944525 17826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:51.944547 17826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:51.945751 17826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:51.945796 17826 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:31:51.945809 17826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:51.945822 17826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:51.945834 17826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:51.945871 17826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:51.947434 17826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39055
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.62.keytab
--principal=oryx/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:51.947701 17826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:51.947935 17826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:31:51.953234 17826 server_base.cc:1034] running on GCE node
W20260925 02:31:51.953176 17831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:51.953176 17834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:51.953178 17832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:51.953699 17826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:51.953977 17826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:51.955124 17826 hybrid_clock.cc:648] HybridClock initialized: now 1790303511955107 us; error 35 us; skew 500 ppm
Sep 25 02:31:51 dist-test-slave-q8mg krb5kdc[17813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303511, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:51.957684 17826 init.cc:377] Logged in from keytab as oryx/127.10.13.62@KRBTEST.COM (short username oryx)
I20260925 02:31:51.958120 17826 webserver.cc:460] Webserver started at http://127.10.13.62:36407/ using document root <none> and password file <none>
I20260925 02:31:51.958351 17826 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:51.958417 17826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:51.958562 17826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:51.959664 17826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "7c54dac4515f4a75845525a447d9bc37"
format_stamp: "Formatted at 2026-09-25 02:31:51 on dist-test-slave-q8mg"
server_key: "6a99fe78c60d02db6964e90b719dac9b"
server_key_iv: "8b13817fc540e044003b550c967a9534"
server_key_version: "encryptionkey@0"
I20260925 02:31:51.960031 17826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "7c54dac4515f4a75845525a447d9bc37"
format_stamp: "Formatted at 2026-09-25 02:31:51 on dist-test-slave-q8mg"
server_key: "6a99fe78c60d02db6964e90b719dac9b"
server_key_iv: "8b13817fc540e044003b550c967a9534"
server_key_version: "encryptionkey@0"
I20260925 02:31:51.961473 17826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 02:31:51.962276 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:51.962464 17826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:51.962572 17826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "7c54dac4515f4a75845525a447d9bc37"
format_stamp: "Formatted at 2026-09-25 02:31:51 on dist-test-slave-q8mg"
server_key: "6a99fe78c60d02db6964e90b719dac9b"
server_key_iv: "8b13817fc540e044003b550c967a9534"
server_key_version: "encryptionkey@0"
I20260925 02:31:51.962642 17826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:51.987102 17826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:51.989545 17826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:51.989714 17826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:51.994505 17826 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:39055
I20260925 02:31:51.994582 17903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:39055 every 8 connection(s)
I20260925 02:31:51.994920 17826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:51.995551 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 17826
I20260925 02:31:51.995678 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:51.995692 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:51.995867 10292 external_mini_cluster.cc:1428] Setting key 40b3d452ec2728f1434ec3215bb786b1
I20260925 02:31:51.998934 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: Bootstrap starting.
I20260925 02:31:51.999576 17904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:51.999855 17904 log.cc:824] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:52.000535 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: No bootstrap required, opened a new log
Sep 25 02:31:51 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:52.002377 17904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54dac4515f4a75845525a447d9bc37" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39055 } }
I20260925 02:31:52.002558 17904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.002622 17904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c54dac4515f4a75845525a447d9bc37, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.002772 17904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54dac4515f4a75845525a447d9bc37" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39055 } }
I20260925 02:31:52.002835 17904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:52.002884 17904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:52.002935 17904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:52.003587 17904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54dac4515f4a75845525a447d9bc37" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39055 } }
I20260925 02:31:52.003705 17904 leader_election.cc:302] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c54dac4515f4a75845525a447d9bc37; no voters:
I20260925 02:31:52.003907 17904 leader_election.cc:288] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:52.003944 17909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:52.004077 17909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [term 1 LEADER]: Becoming Leader. State: Replica: 7c54dac4515f4a75845525a447d9bc37, State: Running, Role: LEADER
I20260925 02:31:52.004181 17904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:52.004160 17909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54dac4515f4a75845525a447d9bc37" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39055 } }
I20260925 02:31:52.004608 17911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c54dac4515f4a75845525a447d9bc37. Latest consensus state: current_term: 1 leader_uuid: "7c54dac4515f4a75845525a447d9bc37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54dac4515f4a75845525a447d9bc37" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39055 } } }
I20260925 02:31:52.004765 17911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:52.004981 17910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c54dac4515f4a75845525a447d9bc37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c54dac4515f4a75845525a447d9bc37" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39055 } } }
I20260925 02:31:52.005118 17910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37 [sys.catalog]: This master's current role is: LEADER
W20260925 02:31:52.006337 17925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:31:52.006436 17925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:31:52.006526 17913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:52.006676 17913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:52.006805 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:51.996401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.10.13.62:39055)
0925 02:31:51.996602 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:51.996612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:51.996649 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:31:51.997376 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:51.997385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:51.997440 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:51.997597 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:51.998245 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:51.999230 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.000088 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.000484 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.000601 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.003027 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.003031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.003031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.003044 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.005193 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.005526 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.005530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.005531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.005627 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.005836 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.005837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.005838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.006212 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.006371 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.006467 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.006606 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260925 02:31:52.008306 17913 catalog_manager.cc:1347] Generated new cluster ID: 99e2d0e399af4b3196f1e52375f11736
I20260925 02:31:52.008347 17913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:52.027017 17913 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:52.027570 17913 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:52.031987 17913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: Generated new TSK 0
I20260925 02:31:52.032161 17913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab.
Entry for principal oryx/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab.
I20260925 02:31:52.063988 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39055
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab
--principal=oryx/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:52.150161 17930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:52.150357 17930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:52.150380 17930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:52.151536 17930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:52.151577 17930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:52.151597 17930 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:52.151639 17930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:52.153239 17930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab
--principal=oryx/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:39055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:52.153488 17930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:52.153723 17930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:52.158823 17935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:52.158874 17938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:52.158830 17936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:52.159129 17930 server_base.cc:1034] running on GCE node
I20260925 02:31:52.159340 17930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:52.159569 17930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:52.160717 17930 hybrid_clock.cc:648] HybridClock initialized: now 1790303512160704 us; error 32 us; skew 500 ppm
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303512, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:52.163326 17930 init.cc:377] Logged in from keytab as oryx/127.10.13.1@KRBTEST.COM (short username oryx)
I20260925 02:31:52.163743 17930 webserver.cc:460] Webserver started at http://127.10.13.1:41397/ using document root <none> and password file <none>
I20260925 02:31:52.163954 17930 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:52.164011 17930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:52.164105 17930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:52.165256 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "7db62669a5b24a9fa858087dbab15ae1"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "d4be4707c81e5fbcf9feee7416dd61fa"
server_key_iv: "7a63b0227334546308a6391e2584fcc1"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.165598 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "7db62669a5b24a9fa858087dbab15ae1"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "d4be4707c81e5fbcf9feee7416dd61fa"
server_key_iv: "7a63b0227334546308a6391e2584fcc1"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.167033 17930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:52.167770 17945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.167936 17930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:52.168035 17930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "7db62669a5b24a9fa858087dbab15ae1"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "d4be4707c81e5fbcf9feee7416dd61fa"
server_key_iv: "7a63b0227334546308a6391e2584fcc1"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.168092 17930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:52.187276 17930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:52.189621 17930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:52.189743 17930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:52.190230 17930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:52.190276 17930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.190304 17930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:52.190346 17930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.197674 17930 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:35349
I20260925 02:31:52.197762 18060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:35349 every 8 connection(s)
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303512, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:52.198150 17930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:52.199221 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 17930
I20260925 02:31:52.199329 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:52.199536 10292 external_mini_cluster.cc:1428] Setting key fe946d2de2347596d3d4c45e3cf74bd0
I20260925 02:31:52.201880 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.190830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:52627 (local address 127.10.13.62:39055)
0925 02:31:52.190889 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.190891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.191156 (+ 265us) server_negotiation.cc:407] Connection header received
0925 02:31:52.191723 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.191724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.191738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.191803 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.192799 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.193665 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.195429 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.195748 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.195808 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.198552 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.198554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.198555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.198563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.200254 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.200768 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.200769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.200769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.200823 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.201132 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.201133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.201134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.201241 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.201297 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.201635 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.201746 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:31:52.202066 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.190831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39055 (local address 127.10.13.1:52627)
0925 02:31:52.191015 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.191053 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.191610 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.191927 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.191933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.192358 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:52.192676 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.192685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.193768 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.193770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.195316 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.195320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.195838 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.195839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.196016 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.196625 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:52.196637 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:52.198462 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:52.200339 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.200347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.200350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.200692 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.200906 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.200907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.200908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.201060 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.201305 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:52.201311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:52.201486 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.201611 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.201802 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260925 02:31:52.202606 18061 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39055
I20260925 02:31:52.202690 18061 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:52.202919 18061 heartbeater.cc:511] Master 127.10.13.62:39055 requested a full tablet report, sending...
I20260925 02:31:52.203533 17857 ts_manager.cc:194] Registered new tserver with Master: 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349)
I20260925 02:31:52.204517 17857 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:52627
WARNING: no policy specified for oryx/127.10.13.2@KRBTEST.COM; defaulting to no policy
I20260925 02:31:52.209873 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.203817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:58013 (local address 127.10.13.62:39055)
0925 02:31:52.203854 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.203856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.203864 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:52.203926 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.203927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.203937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.203991 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.204328 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.205185 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.205990 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.206332 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.206416 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.207281 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.207283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.207283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.207291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.208852 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.209190 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.209192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.209192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.209259 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.209441 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.209442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.209442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.209576 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.209645 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.209752 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.209814 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:31:52.209879 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.203660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39055 (local address 127.10.13.1:58013)
0925 02:31:52.203809 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.203821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.203874 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.203994 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.203994 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.204191 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:52.204249 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.204250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.205297 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.205298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.205912 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.205915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.206412 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.206412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.206464 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.206923 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:52.206927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:52.207197 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:52.208952 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.208953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.208953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.209118 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.209329 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.209329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.209330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.209381 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.209647 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:52.209648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:52.209684 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.209735 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.209812 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab.
Entry for principal oryx/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab.
I20260925 02:31:52.227371 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39055
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab
--principal=oryx/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:52.309973 18070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:52.310195 18070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:52.310222 18070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:52.311355 18070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:52.311393 18070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:52.311419 18070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:52.311437 18070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:52.312934 18070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab
--principal=oryx/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:39055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:52.313122 18070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:52.313308 18070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:52.318457 18078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:52.318497 18075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:52.318499 18070 server_base.cc:1034] running on GCE node
W20260925 02:31:52.318463 18076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:52.318899 18070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:52.319192 18070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:52.320353 18070 hybrid_clock.cc:648] HybridClock initialized: now 1790303512320331 us; error 40 us; skew 500 ppm
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303512, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:52.322988 18070 init.cc:377] Logged in from keytab as oryx/127.10.13.2@KRBTEST.COM (short username oryx)
I20260925 02:31:52.323442 18070 webserver.cc:460] Webserver started at http://127.10.13.2:36075/ using document root <none> and password file <none>
I20260925 02:31:52.323670 18070 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:52.323734 18070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:52.323880 18070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:52.325011 18070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "8dc29e84708946a9b959218b2fa95a5c"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "b55d876173d408a738a22c0ed9853d69"
server_key_iv: "4cde3e51952fbe59f0a0fc8351537794"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.325374 18070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "8dc29e84708946a9b959218b2fa95a5c"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "b55d876173d408a738a22c0ed9853d69"
server_key_iv: "4cde3e51952fbe59f0a0fc8351537794"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.326720 18070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:52.327525 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.327710 18070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:52.327811 18070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "8dc29e84708946a9b959218b2fa95a5c"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "b55d876173d408a738a22c0ed9853d69"
server_key_iv: "4cde3e51952fbe59f0a0fc8351537794"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.327869 18070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:52.346884 18070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:52.349289 18070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:52.349454 18070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:52.350035 18070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:52.350098 18070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.350133 18070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:52.350174 18070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303512, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:52.357792 18070 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:35003
I20260925 02:31:52.357874 18200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:35003 every 8 connection(s)
I20260925 02:31:52.358234 18070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:52.360214 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.350504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:52877 (local address 127.10.13.62:39055)
0925 02:31:52.350585 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.350588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.350891 (+ 303us) server_negotiation.cc:407] Connection header received
0925 02:31:52.351721 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.351723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.351736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.351781 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.352721 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.353316 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.354460 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.354661 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.354736 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.357267 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.357268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.357269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.357276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.358940 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.359405 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.359406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.359407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.359489 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.359731 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.359732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.359732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.359841 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.359893 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.360087 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.360152 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:31:52.360390 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.350498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39055 (local address 127.10.13.2:52877)
0925 02:31:52.350704 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.350756 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.351609 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.351872 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.351877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.352289 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:52.352628 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.352636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.353422 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.353424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.354363 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.354366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.354759 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.354760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.354919 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.355500 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:52.355508 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:52.357165 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:52.359016 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.359020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.359022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.359322 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.359570 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.359571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.359571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.359676 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.359885 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:52.359888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:52.359995 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.360072 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.360187 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260925 02:31:52.360903 18201 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39055
I20260925 02:31:52.360988 18201 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:52.361182 18201 heartbeater.cc:511] Master 127.10.13.62:39055 requested a full tablet report, sending...
I20260925 02:31:52.361567 17857 ts_manager.cc:194] Registered new tserver with Master: 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003)
I20260925 02:31:52.362071 17857 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:52877
I20260925 02:31:52.362571 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 18070
I20260925 02:31:52.362689 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:52.362901 10292 external_mini_cluster.cc:1428] Setting key 9f77ad4b59fe228d12880624f3af1743
I20260925 02:31:52.368283 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.362063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:56553 (local address 127.10.13.62:39055)
0925 02:31:52.362215 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.362217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.362224 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:31:52.362246 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.362246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.362256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.362304 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.362731 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.363350 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.364343 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.364536 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.364593 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.365666 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.365668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.365668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.365676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.367121 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.367525 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.367526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.367527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.367574 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.367775 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.367775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.367775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.367902 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.367949 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.368152 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.368220 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 02:31:52.368358 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.361979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39055 (local address 127.10.13.2:56553)
0925 02:31:52.362133 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.362146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.362201 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.362307 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.362308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.362528 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:52.362613 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.362615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.363441 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.363442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.364267 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.364271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.364610 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.364611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.364685 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.365227 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:52.365232 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:52.365583 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:52.367203 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.367205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.367205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.367448 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.367635 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.367636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.367637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.367713 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.367964 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:52.367965 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:52.368018 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.368098 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.368221 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab.
Entry for principal oryx/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab.
I20260925 02:31:52.389698 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39055
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab
--principal=oryx/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:52.475675 18210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:52.475889 18210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:52.475914 18210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:52.477155 18210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:52.477201 18210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:52.477223 18210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:31:52.477267 18210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:52.478797 18210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab
--principal=oryx/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.13.62:39055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:52.479056 18210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:52.479310 18210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:52.484593 18218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:52.484613 18210 server_base.cc:1034] running on GCE node
W20260925 02:31:52.484685 18216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:52.484596 18215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:52.485139 18210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:52.485405 18210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:52.486586 18210 hybrid_clock.cc:648] HybridClock initialized: now 1790303512486575 us; error 31 us; skew 500 ppm
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303512, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:52.489212 18210 init.cc:377] Logged in from keytab as oryx/127.10.13.3@KRBTEST.COM (short username oryx)
I20260925 02:31:52.489759 18210 webserver.cc:460] Webserver started at http://127.10.13.3:44313/ using document root <none> and password file <none>
I20260925 02:31:52.490005 18210 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:52.490074 18210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:52.490223 18210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:52.491343 18210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "da81a22a07c94a3c98232dad218f47bb"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "510e4adac42998f2aadde1c1df9d7c11"
server_key_iv: "6ba0097e5ea6cd64571cff0c3f1061b1"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.491717 18210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "da81a22a07c94a3c98232dad218f47bb"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "510e4adac42998f2aadde1c1df9d7c11"
server_key_iv: "6ba0097e5ea6cd64571cff0c3f1061b1"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.493218 18210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:52.494063 18225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.494237 18210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:52.494350 18210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "da81a22a07c94a3c98232dad218f47bb"
format_stamp: "Formatted at 2026-09-25 02:31:52 on dist-test-slave-q8mg"
server_key: "510e4adac42998f2aadde1c1df9d7c11"
server_key_iv: "6ba0097e5ea6cd64571cff0c3f1061b1"
server_key_version: "encryptionkey@0"
I20260925 02:31:52.494426 18210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:52.517122 18210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:52.519177 18210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:52.519347 18210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:52.519948 18210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:52.520006 18210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.520056 18210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:52.520123 18210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303512, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:52.527359 18210 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:41495
I20260925 02:31:52.527725 18210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:31:52.527781 18340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:41495 every 8 connection(s)
I20260925 02:31:52.529769 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.520371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:38747 (local address 127.10.13.62:39055)
0925 02:31:52.520424 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.520427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.520640 (+ 213us) server_negotiation.cc:407] Connection header received
0925 02:31:52.521258 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.521260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.521274 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.521338 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.522051 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.522562 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.523636 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.523884 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.523957 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.526922 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.526924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.526924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.526932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.528495 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.528857 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.528858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.528858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.528976 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.529220 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.529222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.529222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.529347 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.529414 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.529639 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.529705 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:31:52.529922 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.520360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39055 (local address 127.10.13.3:38747)
0925 02:31:52.520520 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.520558 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.521153 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.521413 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.521417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.521738 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:52.521949 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.521954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.522659 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.522660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.523542 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.523545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.523978 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.523979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.524115 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.524623 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:52.524633 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:52.526828 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:52.528571 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.528575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.528577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.528792 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.529035 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.529035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.529036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.529135 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.529411 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:52.529415 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:52.529530 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.529612 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.529733 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260925 02:31:52.530458 18341 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39055
I20260925 02:31:52.530573 18341 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:52.530773 18341 heartbeater.cc:511] Master 127.10.13.62:39055 requested a full tablet report, sending...
I20260925 02:31:52.531153 17857 ts_manager.cc:194] Registered new tserver with Master: da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495)
I20260925 02:31:52.531565 17857 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:38747
I20260925 02:31:52.535521 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 18210
I20260925 02:31:52.535624 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:52.535833 10292 external_mini_cluster.cc:1428] Setting key 7b2460f0ee03b2d880f7cbebf5b7563b
I20260925 02:31:52.536723 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:31:52.537526 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.531622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:57923 (local address 127.10.13.62:39055)
0925 02:31:52.531729 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.531731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.531746 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:31:52.531867 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.531868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.531880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.531950 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.532494 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.533067 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.533714 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.533892 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.533965 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.534807 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.534808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.534809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.534814 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.536403 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.536729 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.536730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.536731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.536786 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.537025 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.537026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.537027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.537137 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.537209 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.537358 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.537433 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:31:52.537554 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.531545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39055 (local address 127.10.13.3:57923)
0925 02:31:52.531694 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.531705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.531772 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.532109 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.532110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.532327 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:52.532415 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.532418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.533160 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.533161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.533650 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.533652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.533973 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.533974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.534065 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.534462 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:52.534467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:52.534737 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:52.536484 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.536486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.536486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.536659 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.536892 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.536893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.536894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.536952 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.537201 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:52.537202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:52.537248 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.537331 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.537436 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303512, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:31:52.558693 17907 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:52.553160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.10.13.62:39055)
0925 02:31:52.553224 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.553226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.553267 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:31:52.553294 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.553295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.553305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.553348 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.553917 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.554447 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.555182 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.555368 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.555423 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.558596 (+ 3173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38958: BlockingRecv error: recv got EOF from 127.0.0.1:38958 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303512, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:52.566710 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.559511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.10.13.62:39055)
0925 02:31:52.559554 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.559556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.559608 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:31:52.559663 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.559663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.559672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.559725 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.560156 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.560618 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.561495 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.561691 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.561752 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.563940 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.563941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.563941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.563948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.565522 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.565859 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.565860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.565860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.565914 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.566190 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.566191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.566191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.566318 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.566416 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.566548 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.566609 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:31:52.568398 17857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:52.570205 17857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:52.577250 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.572816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:35349 (local address 127.0.0.1:39242)
0925 02:31:52.573264 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.573275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.573320 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.574008 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.574009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.574012 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:52.574071 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.574072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.575182 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.575183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.576094 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.576097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.576812 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.576813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.576944 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.576967 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.577074 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.577134 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":96,"threads_started":1}
I20260925 02:31:52.577358 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.572812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:41495 (local address 127.0.0.1:46904)
0925 02:31:52.572996 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.573039 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.573107 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.573412 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.573416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.573418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:52.573481 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.573482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.574762 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.574763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.576114 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.576117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.577122 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.577124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.577193 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.577198 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.577232 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.577280 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":9,"thread_start_us":124,"threads_started":1}
I20260925 02:31:52.577463 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.573087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:35003 (local address 127.0.0.1:42630)
0925 02:31:52.573304 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.573314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.573362 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.573709 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.573710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.573712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:52.573777 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.573779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.574686 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.574688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.576113 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.576117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.577108 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.577109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.577162 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.577166 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.577226 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.577265 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":94,"thread_start_us":74,"threads_started":1}
I20260925 02:31:52.577454 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.573097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.10.13.1:35349)
0925 02:31:52.573769 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.573773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.573795 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:31:52.573837 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.573840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.573855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.573936 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:52.574154 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.575091 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.576180 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.576703 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.576806 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.576916 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.577307 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.577354 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":80,"threads_started":1}
I20260925 02:31:52.577843 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.572925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.10.13.3:41495)
0925 02:31:52.573135 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.573140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.573159 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:31:52.573203 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.573205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.573236 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.573331 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:52.573557 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.574692 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.576222 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.577017 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.577598 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.577684 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.577709 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.577749 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260925 02:31:52.578248 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.573284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.10.13.2:35003)
0925 02:31:52.573474 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.573478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.573492 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:31:52.573525 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.573527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.573537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.573646 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:52.573878 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.574596 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.576221 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.577024 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.577097 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.577172 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.577244 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.577283 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260925 02:31:52.578188 17995 tablet_service.cc:1467] Processing CreateTablet for tablet bc33e7fe56ef4399adbf681bd453381a (DEFAULT_TABLE table=test-table [id=aa4be99d606c447da0116d2c94269d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:52.578572 17995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc33e7fe56ef4399adbf681bd453381a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.578552 18275 tablet_service.cc:1467] Processing CreateTablet for tablet bc33e7fe56ef4399adbf681bd453381a (DEFAULT_TABLE table=test-table [id=aa4be99d606c447da0116d2c94269d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:52.578909 18275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc33e7fe56ef4399adbf681bd453381a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.581384 18135 tablet_service.cc:1467] Processing CreateTablet for tablet bc33e7fe56ef4399adbf681bd453381a (DEFAULT_TABLE table=test-table [id=aa4be99d606c447da0116d2c94269d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:52.581775 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc33e7fe56ef4399adbf681bd453381a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.581781 18369 tablet_bootstrap.cc:492] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: Bootstrap starting.
I20260925 02:31:52.582329 18369 tablet_bootstrap.cc:654] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.582661 18369 log.cc:824] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:52.583442 18369 tablet_bootstrap.cc:492] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: No bootstrap required, opened a new log
I20260925 02:31:52.583539 18369 ts_tablet_manager.cc:1397] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:52.584957 18372 tablet_bootstrap.cc:492] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Bootstrap starting.
I20260925 02:31:52.585080 18369 raft_consensus.cc:348] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.585246 18369 raft_consensus.cc:374] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.585290 18369 raft_consensus.cc:729] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da81a22a07c94a3c98232dad218f47bb, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.585427 18369 consensus_queue.cc:260] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.585594 18370 tablet_bootstrap.cc:492] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: Bootstrap starting.
I20260925 02:31:52.585731 18369 ts_tablet_manager.cc:1428] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:31:52.585873 18341 heartbeater.cc:503] Master 127.10.13.62:39055 was elected leader, sending a full tablet report...
I20260925 02:31:52.585911 18372 tablet_bootstrap.cc:654] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.586170 18372 log.cc:824] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:52.586266 18370 tablet_bootstrap.cc:654] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.586606 18370 log.cc:824] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:52.586946 18372 tablet_bootstrap.cc:492] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: No bootstrap required, opened a new log
I20260925 02:31:52.587028 18372 ts_tablet_manager.cc:1397] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:52.587371 18370 tablet_bootstrap.cc:492] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: No bootstrap required, opened a new log
I20260925 02:31:52.587450 18370 ts_tablet_manager.cc:1397] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:31:52.588836 18372 raft_consensus.cc:348] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.589066 18372 raft_consensus.cc:374] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.589134 18372 raft_consensus.cc:729] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc29e84708946a9b959218b2fa95a5c, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.589315 18372 consensus_queue.cc:260] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.589498 18370 raft_consensus.cc:348] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.589629 18372 ts_tablet_manager.cc:1428] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 02:31:52.589659 18370 raft_consensus.cc:374] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.589699 18370 raft_consensus.cc:729] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db62669a5b24a9fa858087dbab15ae1, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.589704 18201 heartbeater.cc:503] Master 127.10.13.62:39055 was elected leader, sending a full tablet report...
I20260925 02:31:52.589812 18370 consensus_queue.cc:260] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.590085 18370 ts_tablet_manager.cc:1428] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:31:52.590229 18061 heartbeater.cc:503] Master 127.10.13.62:39055 was elected leader, sending a full tablet report...
W20260925 02:31:52.608830 18202 tablet.cc:2369] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:52.621910 18373 raft_consensus.cc:482] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:52.622043 18373 raft_consensus.cc:504] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.622400 18373 leader_election.cc:288] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003)
I20260925 02:31:52.626288 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.622741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:54893 (local address 127.10.13.1:35349)
0925 02:31:52.622795 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.622798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.622808 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:52.622841 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.622842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.622853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.622915 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:52.623105 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.623862 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.625339 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.625896 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.625972 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.625983 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.626174 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.626217 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:31:52.626309 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.622600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:35349 (local address 127.10.13.3:54893)
0925 02:31:52.622668 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.622683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.622761 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.622923 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.622924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.622926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:52.623005 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.623007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.623939 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.623939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.625251 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.625254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.625984 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.625986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.626088 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.626092 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.626164 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.626210 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":22}
I20260925 02:31:52.626359 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.622824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:33155 (local address 127.10.13.2:35003)
0925 02:31:52.622865 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.622867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.623014 (+ 147us) server_negotiation.cc:407] Connection header received
0925 02:31:52.623061 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.623062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.623074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.623133 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:52.623271 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.623949 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.625339 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.625896 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.625972 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.625983 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.626239 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.626285 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:31:52.626585 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.622716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:35003 (local address 127.10.13.3:33155)
0925 02:31:52.622924 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.622936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.622994 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.623145 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.623146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.623147 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:52.623201 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.623202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.624018 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.624018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.625251 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.625254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.626043 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.626044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.626146 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.626151 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.626456 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.626504 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260925 02:31:52.626691 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc33e7fe56ef4399adbf681bd453381a" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1" is_pre_election: true
I20260925 02:31:52.626878 18015 raft_consensus.cc:2454] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 0.
I20260925 02:31:52.626958 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc33e7fe56ef4399adbf681bd453381a" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc29e84708946a9b959218b2fa95a5c" is_pre_election: true
I20260925 02:31:52.627130 18155 raft_consensus.cc:2454] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 0.
I20260925 02:31:52.627128 18226 leader_election.cc:302] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db62669a5b24a9fa858087dbab15ae1, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:52.627251 18373 raft_consensus.cc:2781] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:52.627315 18373 raft_consensus.cc:482] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:52.627341 18373 raft_consensus.cc:3037] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:52.628211 18373 raft_consensus.cc:504] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.628345 18373 leader_election.cc:288] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 election: Requested vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003)
I20260925 02:31:52.628546 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc33e7fe56ef4399adbf681bd453381a" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc29e84708946a9b959218b2fa95a5c"
I20260925 02:31:52.628563 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc33e7fe56ef4399adbf681bd453381a" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1"
I20260925 02:31:52.628631 18155 raft_consensus.cc:3037] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:52.628639 18015 raft_consensus.cc:3037] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:52.629392 18015 raft_consensus.cc:2454] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 1.
I20260925 02:31:52.629410 18155 raft_consensus.cc:2454] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 1.
I20260925 02:31:52.629549 18226 leader_election.cc:302] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc29e84708946a9b959218b2fa95a5c, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:52.629678 18373 raft_consensus.cc:2781] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:52.629848 18373 raft_consensus.cc:686] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 1 LEADER]: Becoming Leader. State: Replica: da81a22a07c94a3c98232dad218f47bb, State: Running, Role: LEADER
I20260925 02:31:52.629956 18373 consensus_queue.cc:237] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } }
I20260925 02:31:52.630847 17858 catalog_manager.cc:5495] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb reported cstate change: term changed from 0 to 1, leader changed from <none> to da81a22a07c94a3c98232dad218f47bb (127.10.13.3). New cstate: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } health_report { overall_health: UNKNOWN } } }
I20260925 02:31:52.642869 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.640379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.10.13.3:41495)
0925 02:31:52.640429 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.640431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.640441 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:52.640528 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.640529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.640556 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.640632 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:52.640780 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.641321 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.642081 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.642279 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.642351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.642539 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:52.642648 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:52.642739 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.642760 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.642779 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:52.644845 18155 raft_consensus.cc:1261] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Refusing update from remote peer da81a22a07c94a3c98232dad218f47bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:52.644909 18015 raft_consensus.cc:1261] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Refusing update from remote peer da81a22a07c94a3c98232dad218f47bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:52.645264 18373 consensus_queue.cc:1035] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:52.645412 18379 consensus_queue.cc:1035] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:52.647817 18382 mvcc.cc:205] Tried to move back new op lower bound from 7333083187790413824 to 7333083187733082112. Current Snapshot: MvccSnapshot[applied={T|T < 7333083187790413824}]
I20260925 02:31:52.650318 17858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:52.650420 17858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:52.651292 17858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: Sending DeleteTablet for 3 replicas of tablet bc33e7fe56ef4399adbf681bd453381a
I20260925 02:31:52.651644 17995 tablet_service.cc:1514] Processing DeleteTablet for tablet bc33e7fe56ef4399adbf681bd453381a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:52 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39242
I20260925 02:31:52.651674 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet bc33e7fe56ef4399adbf681bd453381a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:52 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:42630
I20260925 02:31:52.651942 18392 tablet_replica.cc:330] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: stopping tablet replica
I20260925 02:31:52.651929 18275 tablet_service.cc:1514] Processing DeleteTablet for tablet bc33e7fe56ef4399adbf681bd453381a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:52 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:46904
I20260925 02:31:52.651978 17858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:31:52.652046 18392 raft_consensus.cc:2229] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 02:31:52.652115 17858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:52.652122 18394 tablet_replica.cc:330] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: stopping tablet replica
I20260925 02:31:52.652153 18392 raft_consensus.cc:2258] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:52.652204 18394 raft_consensus.cc:2229] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:52.652659 18393 tablet_replica.cc:330] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: stopping tablet replica
I20260925 02:31:52.652686 18392 ts_tablet_manager.cc:1905] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:52.652765 18393 raft_consensus.cc:2229] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:52.652905 18393 raft_consensus.cc:2258] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:52.653724 17995 tablet_service.cc:1467] Processing CreateTablet for tablet cd53dd59c4ac4d37a1903d66f2995663 (DEFAULT_TABLE table=test-table [id=f9de47865993441db54ceb4b64d7c500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:52.653712 18275 tablet_service.cc:1467] Processing CreateTablet for tablet cd53dd59c4ac4d37a1903d66f2995663 (DEFAULT_TABLE table=test-table [id=f9de47865993441db54ceb4b64d7c500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:52.653828 17995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd53dd59c4ac4d37a1903d66f2995663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.653828 18275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd53dd59c4ac4d37a1903d66f2995663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.654764 18392 ts_tablet_manager.cc:1918] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:52.654830 18392 log.cc:1196] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/bc33e7fe56ef4399adbf681bd453381a
I20260925 02:31:52.655083 18392 ts_tablet_manager.cc:1939] T bc33e7fe56ef4399adbf681bd453381a P 7db62669a5b24a9fa858087dbab15ae1: Deleting consensus metadata
I20260925 02:31:52.655158 18370 tablet_bootstrap.cc:492] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: Bootstrap starting.
I20260925 02:31:52.655231 18135 tablet_service.cc:1467] Processing CreateTablet for tablet cd53dd59c4ac4d37a1903d66f2995663 (DEFAULT_TABLE table=test-table [id=f9de47865993441db54ceb4b64d7c500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:52.655350 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd53dd59c4ac4d37a1903d66f2995663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.655546 18370 tablet_bootstrap.cc:654] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.652311 18394 raft_consensus.cc:2258] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:52.656152 18370 tablet_bootstrap.cc:492] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: No bootstrap required, opened a new log
I20260925 02:31:52.656190 18370 ts_tablet_manager.cc:1397] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:52.656322 18370 raft_consensus.cc:348] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.656383 18370 raft_consensus.cc:374] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.656404 18370 raft_consensus.cc:729] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db62669a5b24a9fa858087dbab15ae1, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.656395 18372 tablet_bootstrap.cc:492] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: Bootstrap starting.
I20260925 02:31:52.656423 18369 tablet_bootstrap.cc:492] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: Bootstrap starting.
I20260925 02:31:52.656447 18370 consensus_queue.cc:260] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.656548 18370 ts_tablet_manager.cc:1428] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.656836 18372 tablet_bootstrap.cc:654] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.656836 18369 tablet_bootstrap.cc:654] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.657533 17842 catalog_manager.cc:4841] TS 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349): tablet bc33e7fe56ef4399adbf681bd453381a (table test-table [id=aa4be99d606c447da0116d2c94269d0b]) successfully deleted
I20260925 02:31:52.657559 18393 ts_tablet_manager.cc:1905] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:52.657747 18372 tablet_bootstrap.cc:492] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: No bootstrap required, opened a new log
I20260925 02:31:52.657797 18372 ts_tablet_manager.cc:1397] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:52.657943 18372 raft_consensus.cc:348] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.658026 18372 raft_consensus.cc:374] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.658051 18372 raft_consensus.cc:729] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc29e84708946a9b959218b2fa95a5c, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.658100 18372 consensus_queue.cc:260] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.658190 18369 tablet_bootstrap.cc:492] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: No bootstrap required, opened a new log
I20260925 02:31:52.658203 18372 ts_tablet_manager.cc:1428] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:52.658232 18369 ts_tablet_manager.cc:1397] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:31:52.658372 18369 raft_consensus.cc:348] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.658447 18369 raft_consensus.cc:374] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.658481 18369 raft_consensus.cc:729] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da81a22a07c94a3c98232dad218f47bb, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.658532 18369 consensus_queue.cc:260] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.658624 18369 ts_tablet_manager.cc:1428] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.659103 18393 ts_tablet_manager.cc:1918] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:52.659174 18393 log.cc:1196] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/bc33e7fe56ef4399adbf681bd453381a
I20260925 02:31:52.659437 18393 ts_tablet_manager.cc:1939] T bc33e7fe56ef4399adbf681bd453381a P 8dc29e84708946a9b959218b2fa95a5c: Deleting consensus metadata
I20260925 02:31:52.659826 17842 catalog_manager.cc:4841] TS 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003): tablet bc33e7fe56ef4399adbf681bd453381a (table test-table [id=aa4be99d606c447da0116d2c94269d0b]) successfully deleted
I20260925 02:31:52.660127 18394 ts_tablet_manager.cc:1905] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:52.661736 18394 ts_tablet_manager.cc:1918] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:52.661811 18394 log.cc:1196] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/bc33e7fe56ef4399adbf681bd453381a
I20260925 02:31:52.662072 18394 ts_tablet_manager.cc:1939] T bc33e7fe56ef4399adbf681bd453381a P da81a22a07c94a3c98232dad218f47bb: Deleting consensus metadata
I20260925 02:31:52.662503 17844 catalog_manager.cc:4841] TS da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495): tablet bc33e7fe56ef4399adbf681bd453381a (table test-table [id=aa4be99d606c447da0116d2c94269d0b]) successfully deleted
W20260925 02:31:52.699301 18062 tablet.cc:2369] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:31:52.770205 18389 raft_consensus.cc:482] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:52.770301 18389 raft_consensus.cc:504] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.770443 18389 leader_election.cc:288] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003)
I20260925 02:31:52.770730 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd53dd59c4ac4d37a1903d66f2995663" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1" is_pre_election: true
I20260925 02:31:52.770841 18015 raft_consensus.cc:2454] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 0.
I20260925 02:31:52.770779 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd53dd59c4ac4d37a1903d66f2995663" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc29e84708946a9b959218b2fa95a5c" is_pre_election: true
I20260925 02:31:52.770886 18155 raft_consensus.cc:2454] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 0.
I20260925 02:31:52.771040 18226 leader_election.cc:302] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db62669a5b24a9fa858087dbab15ae1, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:52.771171 18389 raft_consensus.cc:2781] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:52.771226 18389 raft_consensus.cc:482] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:52.771258 18389 raft_consensus.cc:3037] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:52.772019 18389 raft_consensus.cc:504] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.772162 18389 leader_election.cc:288] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 election: Requested vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003)
I20260925 02:31:52.772356 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd53dd59c4ac4d37a1903d66f2995663" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1"
I20260925 02:31:52.772356 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd53dd59c4ac4d37a1903d66f2995663" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc29e84708946a9b959218b2fa95a5c"
I20260925 02:31:52.772442 18015 raft_consensus.cc:3037] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:52.772442 18155 raft_consensus.cc:3037] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:52.773244 18015 raft_consensus.cc:2454] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 1.
I20260925 02:31:52.773244 18155 raft_consensus.cc:2454] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 1.
I20260925 02:31:52.773442 18226 leader_election.cc:302] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db62669a5b24a9fa858087dbab15ae1, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:52.773579 18389 raft_consensus.cc:2781] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:52.773655 18389 raft_consensus.cc:686] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 1 LEADER]: Becoming Leader. State: Replica: da81a22a07c94a3c98232dad218f47bb, State: Running, Role: LEADER
I20260925 02:31:52.773710 18389 consensus_queue.cc:237] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.774351 17857 catalog_manager.cc:5495] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb reported cstate change: term changed from 0 to 1, leader changed from <none> to da81a22a07c94a3c98232dad218f47bb (127.10.13.3). New cstate: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } health_report { overall_health: HEALTHY } } }
W20260925 02:31:52.778282 18342 tablet.cc:2369] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:31:52 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303511, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.62@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:52.828943 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.821730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38966 (local address 127.10.13.62:39055)
0925 02:31:52.821811 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:52.821813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:52.821924 (+ 111us) server_negotiation.cc:407] Connection header received
0925 02:31:52.822072 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:52.822073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:52.822086 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:52.822149 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:52.822632 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.823129 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.823835 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.824025 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.824107 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.826351 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:52.826352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:52.826352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:52.826360 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:52.827705 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.828126 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.828127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.828128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.828188 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:52.828372 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:52.828373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:52.828374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:52.828492 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:52.828560 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:52.828716 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 02:31:52.828801 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:31:52.829007 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:52.821657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39055 (local address 127.0.0.1:38966)
0925 02:31:52.821851 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:52.821861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:52.821945 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:52.822165 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:52.822166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:52.822503 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:52.822550 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.822552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.823219 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.823221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.823755 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:52.823756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:52.824110 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:52.824110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:52.824167 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:52.824719 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:52.824724 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:52.826258 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:52.827783 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.827792 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.827795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.828049 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.828252 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:52.828253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:52.828253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:52.828312 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:52.828560 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:52.828570 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:52.828616 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:31:52.828692 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:31:52.828857 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":63,"thread_start_us":100,"threads_started":1}
I20260925 02:31:52.831197 17857 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:38966:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:31:52.831434 17857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:52.832957 18135 tablet_service.cc:1467] Processing CreateTablet for tablet d5a9b854eb024822b89793326aa11537 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4dfe08e47c3d4b88b7c8815fce89dbf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:52.832957 17995 tablet_service.cc:1467] Processing CreateTablet for tablet d5a9b854eb024822b89793326aa11537 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4dfe08e47c3d4b88b7c8815fce89dbf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:52.833125 17995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a9b854eb024822b89793326aa11537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.833127 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a9b854eb024822b89793326aa11537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.834697 18370 tablet_bootstrap.cc:492] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: Bootstrap starting.
I20260925 02:31:52.834767 18275 tablet_service.cc:1467] Processing CreateTablet for tablet d5a9b854eb024822b89793326aa11537 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4dfe08e47c3d4b88b7c8815fce89dbf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:31:52.834909 18275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a9b854eb024822b89793326aa11537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:52.835085 18370 tablet_bootstrap.cc:654] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.835368 18372 tablet_bootstrap.cc:492] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: Bootstrap starting.
I20260925 02:31:52.835752 18372 tablet_bootstrap.cc:654] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.835748 18370 tablet_bootstrap.cc:492] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: No bootstrap required, opened a new log
I20260925 02:31:52.835793 18370 ts_tablet_manager.cc:1397] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:52.835948 18370 raft_consensus.cc:348] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.836058 18370 raft_consensus.cc:374] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.836087 18370 raft_consensus.cc:729] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db62669a5b24a9fa858087dbab15ae1, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.836079 18369 tablet_bootstrap.cc:492] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: Bootstrap starting.
I20260925 02:31:52.836138 18370 consensus_queue.cc:260] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.836267 18370 ts_tablet_manager.cc:1428] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:52.836318 18377 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:52.836418 18369 tablet_bootstrap.cc:654] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:52.836421 18372 tablet_bootstrap.cc:492] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: No bootstrap required, opened a new log
I20260925 02:31:52.836434 18377 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:52.836458 18372 ts_tablet_manager.cc:1397] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:52.836640 18372 raft_consensus.cc:348] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.836736 18372 raft_consensus.cc:374] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.836755 18372 raft_consensus.cc:729] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc29e84708946a9b959218b2fa95a5c, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.836807 18372 consensus_queue.cc:260] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.836920 18372 ts_tablet_manager.cc:1428] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:52.836966 18376 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:52.837062 18376 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:52.837333 18369 tablet_bootstrap.cc:492] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: No bootstrap required, opened a new log
I20260925 02:31:52.837385 18369 ts_tablet_manager.cc:1397] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:52.837515 18369 raft_consensus.cc:348] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.837600 18369 raft_consensus.cc:374] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:52.837620 18369 raft_consensus.cc:729] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da81a22a07c94a3c98232dad218f47bb, State: Initialized, Role: FOLLOWER
I20260925 02:31:52.837670 18369 consensus_queue.cc:260] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:52.837778 18369 ts_tablet_manager.cc:1428] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:52.837795 18389 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:52.837867 18389 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:53.085625 18389 raft_consensus.cc:482] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:53.085723 18389 raft_consensus.cc:504] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.085891 18389 leader_election.cc:288] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003)
I20260925 02:31:53.086205 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a9b854eb024822b89793326aa11537" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1" is_pre_election: true
I20260925 02:31:53.086302 18015 raft_consensus.cc:2454] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 0.
I20260925 02:31:53.086228 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a9b854eb024822b89793326aa11537" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc29e84708946a9b959218b2fa95a5c" is_pre_election: true
I20260925 02:31:53.086346 18155 raft_consensus.cc:2454] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 0.
I20260925 02:31:53.086472 18226 leader_election.cc:302] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db62669a5b24a9fa858087dbab15ae1, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:53.086587 18389 raft_consensus.cc:2781] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:53.086627 18389 raft_consensus.cc:482] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:53.086647 18389 raft_consensus.cc:3037] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.087313 18389 raft_consensus.cc:504] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.087450 18389 leader_election.cc:288] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 election: Requested vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003)
I20260925 02:31:53.087600 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a9b854eb024822b89793326aa11537" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1"
I20260925 02:31:53.087627 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a9b854eb024822b89793326aa11537" candidate_uuid: "da81a22a07c94a3c98232dad218f47bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc29e84708946a9b959218b2fa95a5c"
I20260925 02:31:53.087692 18155 raft_consensus.cc:3037] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.087692 18015 raft_consensus.cc:3037] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.088443 18015 raft_consensus.cc:2454] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 1.
I20260925 02:31:53.088443 18155 raft_consensus.cc:2454] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da81a22a07c94a3c98232dad218f47bb in term 1.
I20260925 02:31:53.088630 18226 leader_election.cc:302] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc29e84708946a9b959218b2fa95a5c, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:53.088744 18389 raft_consensus.cc:2781] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:53.088793 18389 raft_consensus.cc:686] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [term 1 LEADER]: Becoming Leader. State: Replica: da81a22a07c94a3c98232dad218f47bb, State: Running, Role: LEADER
I20260925 02:31:53.088851 18389 consensus_queue.cc:237] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.089186 18373 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: TxnStatusTablet state changed. Reason: New leader da81a22a07c94a3c98232dad218f47bb. Latest consensus state: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:53.089273 18373 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:53.089421 18409 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:31:53.089560 18409 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:31:53.089578 17858 catalog_manager.cc:5495] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb reported cstate change: term changed from 0 to 1, leader changed from <none> to da81a22a07c94a3c98232dad218f47bb (127.10.13.3). New cstate: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } health_report { overall_health: HEALTHY } } }
I20260925 02:31:53.121196 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.118768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.10.13.3:41495)
0925 02:31:53.118816 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.118818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.118829 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:53.118922 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.118923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.118935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.119007 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:53.119158 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.119629 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.120422 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.120635 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.120708 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.120904 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.120945 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.121005 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:53.121087 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:31:53.121110 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:31:53.121203 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.118649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:41495 (local address 127.0.0.1:46922)
0925 02:31:53.118771 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:53.118785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:53.118852 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:53.119019 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:53.119020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:53.119021 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:53.119081 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.119083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.119718 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.119719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.120333 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.120335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.120710 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.120711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.120779 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.120803 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.121007 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.121018 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:31:53.121081 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:31:53.121133 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:31:53.122272 18155 raft_consensus.cc:1261] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Refusing update from remote peer da81a22a07c94a3c98232dad218f47bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:53.122272 18015 raft_consensus.cc:1261] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Refusing update from remote peer da81a22a07c94a3c98232dad218f47bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:53.122524 18389 consensus_queue.cc:1035] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:53.122650 18373 consensus_queue.cc:1035] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:53.123171 18376 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: TxnStatusTablet state changed. Reason: New leader da81a22a07c94a3c98232dad218f47bb. Latest consensus state: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:53.123179 18377 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: TxnStatusTablet state changed. Reason: New leader da81a22a07c94a3c98232dad218f47bb. Latest consensus state: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:53.123258 18376 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:53.123258 18377 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:53.124307 18383 mvcc.cc:205] Tried to move back new op lower bound from 7333083189746995200 to 7333083189612621824. Current Snapshot: MvccSnapshot[applied={T|T < 7333083189746995200}]
I20260925 02:31:53.124498 18376 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:53.124591 18376 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P 8dc29e84708946a9b959218b2fa95a5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:53.124583 18384 mvcc.cc:205] Tried to move back new op lower bound from 7333083189746995200 to 7333083189612621824. Current Snapshot: MvccSnapshot[applied={T|T < 7333083189746995200}]
I20260925 02:31:53.124676 18377 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:53.124745 18377 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P 7db62669a5b24a9fa858087dbab15ae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:31:53.124765 18373 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:53.124840 18373 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:53.125108 18379 tablet_replica.cc:439] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da81a22a07c94a3c98232dad218f47bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } } }
I20260925 02:31:53.125207 18379 tablet_replica.cc:442] T d5a9b854eb024822b89793326aa11537 P da81a22a07c94a3c98232dad218f47bb: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:31:53.130239 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.127629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:36595 (local address 127.10.13.3:41495)
0925 02:31:53.127669 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.127671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.127755 (+ 84us) server_negotiation.cc:407] Connection header received
0925 02:31:53.127820 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.127821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.127828 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.127865 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:53.128062 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.128537 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.129636 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.129821 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.129872 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.130040 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.130067 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.130104 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:53.130157 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 02:31:53.130171 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:31:53.130319 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.127558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:41495 (local address 127.10.13.3:36595)
0925 02:31:53.127691 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:53.127702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:53.127765 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:53.127878 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:53.127879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:53.127880 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:53.127944 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.127946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.128607 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.128609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.129558 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.129561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.129885 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.129887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.129949 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.129966 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.130105 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.130106 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 02:31:53.130154 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:31:53.130199 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260925 02:31:53.132577 18015 raft_consensus.cc:1261] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Refusing update from remote peer da81a22a07c94a3c98232dad218f47bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:53.132611 18155 raft_consensus.cc:1261] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Refusing update from remote peer da81a22a07c94a3c98232dad218f47bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:53.132802 18389 consensus_queue.cc:1035] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:53.132933 18389 consensus_queue.cc:1035] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:53.147197 17856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:53.147320 17856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:53.147994 17856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: Sending DeleteTablet for 3 replicas of tablet cd53dd59c4ac4d37a1903d66f2995663
I20260925 02:31:53.148263 18275 tablet_service.cc:1514] Processing DeleteTablet for tablet cd53dd59c4ac4d37a1903d66f2995663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:46904
I20260925 02:31:53.148278 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet cd53dd59c4ac4d37a1903d66f2995663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:42630
I20260925 02:31:53.148263 17995 tablet_service.cc:1514] Processing DeleteTablet for tablet cd53dd59c4ac4d37a1903d66f2995663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39242
I20260925 02:31:53.148377 18394 tablet_replica.cc:330] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: stopping tablet replica
I20260925 02:31:53.148447 18394 raft_consensus.cc:2229] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:53.148505 18394 raft_consensus.cc:2258] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.148581 18392 tablet_replica.cc:330] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: stopping tablet replica
I20260925 02:31:53.148703 18392 raft_consensus.cc:2229] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:53.148746 18392 raft_consensus.cc:2258] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.148823 18393 tablet_replica.cc:330] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: stopping tablet replica
I20260925 02:31:53.148905 18393 raft_consensus.cc:2229] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:53.148948 18393 raft_consensus.cc:2258] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.151556 18394 ts_tablet_manager.cc:1905] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:53.153012 18393 ts_tablet_manager.cc:1905] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:53.153039 18392 ts_tablet_manager.cc:1905] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:53.154552 18394 ts_tablet_manager.cc:1918] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:53.154623 18394 log.cc:1196] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/cd53dd59c4ac4d37a1903d66f2995663
I20260925 02:31:53.154628 18392 ts_tablet_manager.cc:1918] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:53.154677 18392 log.cc:1196] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/cd53dd59c4ac4d37a1903d66f2995663
I20260925 02:31:53.154853 18394 ts_tablet_manager.cc:1939] T cd53dd59c4ac4d37a1903d66f2995663 P da81a22a07c94a3c98232dad218f47bb: Deleting consensus metadata
I20260925 02:31:53.154874 18392 ts_tablet_manager.cc:1939] T cd53dd59c4ac4d37a1903d66f2995663 P 7db62669a5b24a9fa858087dbab15ae1: Deleting consensus metadata
I20260925 02:31:53.155202 17844 catalog_manager.cc:4841] TS da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495): tablet cd53dd59c4ac4d37a1903d66f2995663 (table test-table [id=f9de47865993441db54ceb4b64d7c500]) successfully deleted
I20260925 02:31:53.155293 17842 catalog_manager.cc:4841] TS 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349): tablet cd53dd59c4ac4d37a1903d66f2995663 (table test-table [id=f9de47865993441db54ceb4b64d7c500]) successfully deleted
I20260925 02:31:53.155424 18393 ts_tablet_manager.cc:1918] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:53.155494 18393 log.cc:1196] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/cd53dd59c4ac4d37a1903d66f2995663
I20260925 02:31:53.155689 18393 ts_tablet_manager.cc:1939] T cd53dd59c4ac4d37a1903d66f2995663 P 8dc29e84708946a9b959218b2fa95a5c: Deleting consensus metadata
I20260925 02:31:53.156008 17842 catalog_manager.cc:4841] TS 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003): tablet cd53dd59c4ac4d37a1903d66f2995663 (table test-table [id=f9de47865993441db54ceb4b64d7c500]) successfully deleted
Sep 25 02:31:53 dist-test-slave-q8mg krb5kdc[17813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:31:53 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:53 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:31:53.175522 17907 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:53.169348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.10.13.62:39055)
0925 02:31:53.169424 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.169427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.169525 (+ 98us) server_negotiation.cc:407] Connection header received
0925 02:31:53.169616 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.169617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.169628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.169699 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:53.170391 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.170921 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.171794 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.172005 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.172076 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.175450 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38978: BlockingRecv error: recv got EOF from 127.0.0.1:38978 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 02:31:53 dist-test-slave-q8mg krb5kdc[17813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM
I20260925 02:31:53.184017 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.176352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.10.13.62:39055)
0925 02:31:53.176420 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.176423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.176435 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:31:53.176515 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.176516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.176528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.176593 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:53.177200 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.177936 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.178674 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.178967 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.179048 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.181290 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:53.181292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:53.181292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:53.181300 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:53.182861 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:53.183227 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:53.183228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:53.183229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:53.183292 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:53.183496 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:53.183497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:53.183497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:53.183650 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:53.183749 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:53.183840 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 02:31:53.183917 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:31:53.185707 17856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:31:53.185899 17856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:53.187439 18275 tablet_service.cc:1467] Processing CreateTablet for tablet c5c909d5a0be4b96bd79c8354e4b7d54 (DEFAULT_TABLE table=test-table [id=1afc6a5bfa8a4203b62fabafeb379a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:53.187443 18135 tablet_service.cc:1467] Processing CreateTablet for tablet c5c909d5a0be4b96bd79c8354e4b7d54 (DEFAULT_TABLE table=test-table [id=1afc6a5bfa8a4203b62fabafeb379a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:53.187594 18275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c909d5a0be4b96bd79c8354e4b7d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:53.187601 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c909d5a0be4b96bd79c8354e4b7d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:53.188655 18372 tablet_bootstrap.cc:492] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: Bootstrap starting.
I20260925 02:31:53.188736 17995 tablet_service.cc:1467] Processing CreateTablet for tablet c5c909d5a0be4b96bd79c8354e4b7d54 (DEFAULT_TABLE table=test-table [id=1afc6a5bfa8a4203b62fabafeb379a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:53.188863 17995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c909d5a0be4b96bd79c8354e4b7d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:53.189057 18369 tablet_bootstrap.cc:492] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: Bootstrap starting.
I20260925 02:31:53.189095 18372 tablet_bootstrap.cc:654] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:53.189363 18369 tablet_bootstrap.cc:654] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:53.189921 18369 tablet_bootstrap.cc:492] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: No bootstrap required, opened a new log
I20260925 02:31:53.189977 18369 ts_tablet_manager.cc:1397] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:53.190104 18369 raft_consensus.cc:348] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.190182 18369 raft_consensus.cc:374] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:53.190203 18369 raft_consensus.cc:729] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da81a22a07c94a3c98232dad218f47bb, State: Initialized, Role: FOLLOWER
I20260925 02:31:53.190230 18370 tablet_bootstrap.cc:492] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: Bootstrap starting.
I20260925 02:31:53.190246 18369 consensus_queue.cc:260] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.190336 18369 ts_tablet_manager.cc:1428] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:53.190665 18370 tablet_bootstrap.cc:654] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:53.191260 18372 tablet_bootstrap.cc:492] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: No bootstrap required, opened a new log
I20260925 02:31:53.191295 18372 ts_tablet_manager.cc:1397] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 02:31:53.191335 18370 tablet_bootstrap.cc:492] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: No bootstrap required, opened a new log
I20260925 02:31:53.191380 18370 ts_tablet_manager.cc:1397] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:53.191406 18372 raft_consensus.cc:348] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.191457 18372 raft_consensus.cc:374] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:53.191478 18372 raft_consensus.cc:729] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc29e84708946a9b959218b2fa95a5c, State: Initialized, Role: FOLLOWER
I20260925 02:31:53.191538 18372 consensus_queue.cc:260] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.191526 18370 raft_consensus.cc:348] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.191591 18370 raft_consensus.cc:374] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:53.191612 18370 raft_consensus.cc:729] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db62669a5b24a9fa858087dbab15ae1, State: Initialized, Role: FOLLOWER
I20260925 02:31:53.191654 18372 ts_tablet_manager.cc:1428] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:53.191669 18370 consensus_queue.cc:260] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.191764 18370 ts_tablet_manager.cc:1428] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:53.387480 18376 raft_consensus.cc:482] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:53.387624 18376 raft_consensus.cc:504] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.388038 18376 leader_election.cc:288] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495)
I20260925 02:31:53.391124 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.388406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:51553 (local address 127.10.13.3:41495)
0925 02:31:53.388458 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.388460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.388503 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:31:53.388610 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.388611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.388621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.388692 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:53.388887 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.389345 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.390399 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.390748 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.390829 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.390840 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:53.391000 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 02:31:53.391036 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:31:53.391168 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.388267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:41495 (local address 127.10.13.2:51553)
0925 02:31:53.388425 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:53.388438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:53.388521 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:53.388714 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:53.388715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:53.388716 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:53.388795 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.388797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.389421 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.389422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.390278 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.390281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.390866 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.390868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.390922 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.390925 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:53.390992 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:31:53.391033 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":64,"thread_start_us":59,"threads_started":1}
I20260925 02:31:53.391194 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.388267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:35349 (local address 127.10.13.2:58141)
0925 02:31:53.388410 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:53.388427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:53.388491 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:53.388667 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:53.388668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:53.388671 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:31:53.388738 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.388741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.389481 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.389482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.390276 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.390278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.390866 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.390868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.390924 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.390928 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:31:53.390990 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:31:53.391032 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260925 02:31:53.391441 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c909d5a0be4b96bd79c8354e4b7d54" candidate_uuid: "8dc29e84708946a9b959218b2fa95a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da81a22a07c94a3c98232dad218f47bb" is_pre_election: true
I20260925 02:31:53.391624 18295 raft_consensus.cc:2454] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc29e84708946a9b959218b2fa95a5c in term 0.
I20260925 02:31:53.391634 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.388348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:58141 (local address 127.10.13.1:35349)
0925 02:31:53.388473 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.388475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.388486 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:53.388571 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.388572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.388584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.388650 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:31:53.388817 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.389349 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.390369 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.390749 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.390822 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.390837 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:53.390999 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 02:31:53.391036 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260925 02:31:53.391849 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c909d5a0be4b96bd79c8354e4b7d54" candidate_uuid: "8dc29e84708946a9b959218b2fa95a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1" is_pre_election: true
I20260925 02:31:53.391934 18015 raft_consensus.cc:2454] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc29e84708946a9b959218b2fa95a5c in term 0.
I20260925 02:31:53.392002 18088 leader_election.cc:302] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc29e84708946a9b959218b2fa95a5c, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:53.392136 18376 raft_consensus.cc:2781] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:53.392179 18376 raft_consensus.cc:482] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:53.392205 18376 raft_consensus.cc:3037] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.392925 18376 raft_consensus.cc:504] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.393077 18376 leader_election.cc:288] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [CANDIDATE]: Term 1 election: Requested vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495)
I20260925 02:31:53.393234 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c909d5a0be4b96bd79c8354e4b7d54" candidate_uuid: "8dc29e84708946a9b959218b2fa95a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1"
I20260925 02:31:53.393251 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c909d5a0be4b96bd79c8354e4b7d54" candidate_uuid: "8dc29e84708946a9b959218b2fa95a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da81a22a07c94a3c98232dad218f47bb"
I20260925 02:31:53.393312 18295 raft_consensus.cc:3037] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.393311 18015 raft_consensus.cc:3037] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.394075 18015 raft_consensus.cc:2454] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc29e84708946a9b959218b2fa95a5c in term 1.
I20260925 02:31:53.394169 18295 raft_consensus.cc:2454] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc29e84708946a9b959218b2fa95a5c in term 1.
I20260925 02:31:53.394271 18086 leader_election.cc:302] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db62669a5b24a9fa858087dbab15ae1, 8dc29e84708946a9b959218b2fa95a5c; no voters:
I20260925 02:31:53.394369 18376 raft_consensus.cc:2781] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:53.394475 18376 raft_consensus.cc:686] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 1 LEADER]: Becoming Leader. State: Replica: 8dc29e84708946a9b959218b2fa95a5c, State: Running, Role: LEADER
I20260925 02:31:53.394548 18376 consensus_queue.cc:237] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.395236 17856 catalog_manager.cc:5495] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2). New cstate: current_term: 1 leader_uuid: "8dc29e84708946a9b959218b2fa95a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } health_report { overall_health: UNKNOWN } } }
I20260925 02:31:53.421695 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.419022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.10.13.2:35003)
0925 02:31:53.419259 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.419260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.419270 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:53.419317 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.419318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.419340 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.419425 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:53.419566 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.420075 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.420804 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.421089 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.421151 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.421278 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.421403 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.421468 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:53.421501 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 02:31:53.421519 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":110,"threads_started":1}
I20260925 02:31:53.422780 18015 raft_consensus.cc:1261] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Refusing update from remote peer 8dc29e84708946a9b959218b2fa95a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:53.422804 18295 raft_consensus.cc:1261] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Refusing update from remote peer 8dc29e84708946a9b959218b2fa95a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:53.423076 18376 consensus_queue.cc:1035] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:53.423210 18438 consensus_queue.cc:1035] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:53.426851 17856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:53.426956 17856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:53.427577 17856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: Sending DeleteTablet for 3 replicas of tablet c5c909d5a0be4b96bd79c8354e4b7d54
I20260925 02:31:53.427841 18275 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c909d5a0be4b96bd79c8354e4b7d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:46904
I20260925 02:31:53.427950 18394 tablet_replica.cc:330] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: stopping tablet replica
I20260925 02:31:53.427951 17995 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c909d5a0be4b96bd79c8354e4b7d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39242
I20260925 02:31:53.427992 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c909d5a0be4b96bd79c8354e4b7d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:42630
I20260925 02:31:53.428015 18394 raft_consensus.cc:2229] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:53.428053 18392 tablet_replica.cc:330] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: stopping tablet replica
I20260925 02:31:53.428062 18394 raft_consensus.cc:2258] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.428066 18393 tablet_replica.cc:330] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: stopping tablet replica
I20260925 02:31:53.428099 18392 raft_consensus.cc:2229] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:53.428114 18393 raft_consensus.cc:2229] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:53.428153 18392 raft_consensus.cc:2258] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.428215 18393 raft_consensus.cc:2258] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.428494 18392 ts_tablet_manager.cc:1905] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:53.428643 17856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:31:53.428793 18393 ts_tablet_manager.cc:1905] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:31:53.428805 17856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:31:53.430346 18394 ts_tablet_manager.cc:1905] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:53.430379 18392 ts_tablet_manager.cc:1918] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:53.430442 18392 log.cc:1196] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/c5c909d5a0be4b96bd79c8354e4b7d54
I20260925 02:31:53.430682 18392 ts_tablet_manager.cc:1939] T c5c909d5a0be4b96bd79c8354e4b7d54 P 7db62669a5b24a9fa858087dbab15ae1: Deleting consensus metadata
I20260925 02:31:53.431028 17842 catalog_manager.cc:4841] TS 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349): tablet c5c909d5a0be4b96bd79c8354e4b7d54 (table test-table [id=1afc6a5bfa8a4203b62fabafeb379a1e]) successfully deleted
I20260925 02:31:53.431578 17995 tablet_service.cc:1467] Processing CreateTablet for tablet 6b812bf8dd3d4a449b7b3b51402969fd (DEFAULT_TABLE table=test-table [id=4ef6eaf3168340afb85fc648ddb3771d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:53.431636 18275 tablet_service.cc:1467] Processing CreateTablet for tablet 6b812bf8dd3d4a449b7b3b51402969fd (DEFAULT_TABLE table=test-table [id=4ef6eaf3168340afb85fc648ddb3771d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:53.431694 17995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b812bf8dd3d4a449b7b3b51402969fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:53.431737 18275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b812bf8dd3d4a449b7b3b51402969fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:53.431960 18394 ts_tablet_manager.cc:1918] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:53.432003 18394 log.cc:1196] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/c5c909d5a0be4b96bd79c8354e4b7d54
I20260925 02:31:53.432204 18394 ts_tablet_manager.cc:1939] T c5c909d5a0be4b96bd79c8354e4b7d54 P da81a22a07c94a3c98232dad218f47bb: Deleting consensus metadata
I20260925 02:31:53.432859 18370 tablet_bootstrap.cc:492] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1: Bootstrap starting.
I20260925 02:31:53.432921 18369 tablet_bootstrap.cc:492] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: Bootstrap starting.
I20260925 02:31:53.433236 18369 tablet_bootstrap.cc:654] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:53.433297 18370 tablet_bootstrap.cc:654] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:53.433517 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 6b812bf8dd3d4a449b7b3b51402969fd (DEFAULT_TABLE table=test-table [id=4ef6eaf3168340afb85fc648ddb3771d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:31:53.434087 18370 tablet_bootstrap.cc:492] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1: No bootstrap required, opened a new log
I20260925 02:31:53.434137 18370 ts_tablet_manager.cc:1397] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:53.434270 18370 raft_consensus.cc:348] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.434335 18370 raft_consensus.cc:374] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:53.434355 18370 raft_consensus.cc:729] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db62669a5b24a9fa858087dbab15ae1, State: Initialized, Role: FOLLOWER
I20260925 02:31:53.434399 18370 consensus_queue.cc:260] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.434507 18370 ts_tablet_manager.cc:1428] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:53.434566 17844 catalog_manager.cc:4841] TS da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495): tablet c5c909d5a0be4b96bd79c8354e4b7d54 (table test-table [id=1afc6a5bfa8a4203b62fabafeb379a1e]) successfully deleted
I20260925 02:31:53.434805 18393 ts_tablet_manager.cc:1918] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:31:53.434866 18393 log.cc:1196] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/c5c909d5a0be4b96bd79c8354e4b7d54
I20260925 02:31:53.435093 18393 ts_tablet_manager.cc:1939] T c5c909d5a0be4b96bd79c8354e4b7d54 P 8dc29e84708946a9b959218b2fa95a5c: Deleting consensus metadata
I20260925 02:31:53.435253 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b812bf8dd3d4a449b7b3b51402969fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:53.435484 17842 catalog_manager.cc:4841] TS 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003): tablet c5c909d5a0be4b96bd79c8354e4b7d54 (table test-table [id=1afc6a5bfa8a4203b62fabafeb379a1e]) successfully deleted
I20260925 02:31:53.436240 18372 tablet_bootstrap.cc:492] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: Bootstrap starting.
I20260925 02:31:53.436583 18369 tablet_bootstrap.cc:492] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: No bootstrap required, opened a new log
I20260925 02:31:53.436615 18369 ts_tablet_manager.cc:1397] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 02:31:53.436636 18372 tablet_bootstrap.cc:654] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:53.436694 18369 raft_consensus.cc:348] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.436738 18369 raft_consensus.cc:374] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:53.436751 18369 raft_consensus.cc:729] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da81a22a07c94a3c98232dad218f47bb, State: Initialized, Role: FOLLOWER
I20260925 02:31:53.436807 18369 consensus_queue.cc:260] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.436918 18369 ts_tablet_manager.cc:1428] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:53.437402 18372 tablet_bootstrap.cc:492] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: No bootstrap required, opened a new log
I20260925 02:31:53.437461 18372 ts_tablet_manager.cc:1397] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:31:53.437600 18372 raft_consensus.cc:348] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.437680 18372 raft_consensus.cc:374] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:53.437708 18372 raft_consensus.cc:729] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc29e84708946a9b959218b2fa95a5c, State: Initialized, Role: FOLLOWER
I20260925 02:31:53.437757 18372 consensus_queue.cc:260] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.437847 18372 ts_tablet_manager.cc:1428] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:53.474187 18376 raft_consensus.cc:482] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:31:53.474318 18376 raft_consensus.cc:504] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.474541 18376 leader_election.cc:288] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495)
I20260925 02:31:53.474745 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b812bf8dd3d4a449b7b3b51402969fd" candidate_uuid: "8dc29e84708946a9b959218b2fa95a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da81a22a07c94a3c98232dad218f47bb" is_pre_election: true
I20260925 02:31:53.474740 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b812bf8dd3d4a449b7b3b51402969fd" candidate_uuid: "8dc29e84708946a9b959218b2fa95a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1" is_pre_election: true
I20260925 02:31:53.474866 18015 raft_consensus.cc:2454] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc29e84708946a9b959218b2fa95a5c in term 0.
I20260925 02:31:53.474867 18295 raft_consensus.cc:2454] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc29e84708946a9b959218b2fa95a5c in term 0.
I20260925 02:31:53.475024 18088 leader_election.cc:302] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc29e84708946a9b959218b2fa95a5c, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:53.475157 18376 raft_consensus.cc:2781] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:31:53.475208 18376 raft_consensus.cc:482] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:31:53.475241 18376 raft_consensus.cc:3037] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.475955 18376 raft_consensus.cc:504] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.476153 18376 leader_election.cc:288] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [CANDIDATE]: Term 1 election: Requested vote from peers 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349), da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495)
I20260925 02:31:53.476280 18015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b812bf8dd3d4a449b7b3b51402969fd" candidate_uuid: "8dc29e84708946a9b959218b2fa95a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db62669a5b24a9fa858087dbab15ae1"
I20260925 02:31:53.476367 18015 raft_consensus.cc:3037] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.476312 18295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b812bf8dd3d4a449b7b3b51402969fd" candidate_uuid: "8dc29e84708946a9b959218b2fa95a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da81a22a07c94a3c98232dad218f47bb"
I20260925 02:31:53.476408 18295 raft_consensus.cc:3037] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:53.477135 18295 raft_consensus.cc:2454] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc29e84708946a9b959218b2fa95a5c in term 1.
I20260925 02:31:53.477169 18015 raft_consensus.cc:2454] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc29e84708946a9b959218b2fa95a5c in term 1.
I20260925 02:31:53.477306 18088 leader_election.cc:302] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc29e84708946a9b959218b2fa95a5c, da81a22a07c94a3c98232dad218f47bb; no voters:
I20260925 02:31:53.477425 18376 raft_consensus.cc:2781] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:53.477474 18376 raft_consensus.cc:686] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 1 LEADER]: Becoming Leader. State: Replica: 8dc29e84708946a9b959218b2fa95a5c, State: Running, Role: LEADER
I20260925 02:31:53.477525 18376 consensus_queue.cc:237] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } }
I20260925 02:31:53.478060 17856 catalog_manager.cc:5495] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2). New cstate: current_term: 1 leader_uuid: "8dc29e84708946a9b959218b2fa95a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dc29e84708946a9b959218b2fa95a5c" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 } health_report { overall_health: UNKNOWN } } }
I20260925 02:31:53.491359 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.488622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:60431 (local address 127.10.13.3:41495)
0925 02:31:53.488671 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.488673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.488773 (+ 100us) server_negotiation.cc:407] Connection header received
0925 02:31:53.488805 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.488805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.488817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.488915 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:53.489075 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.489750 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.490534 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.490806 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.490881 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.491114 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.491148 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.491207 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:53.491262 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 02:31:53.491282 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:31:53.491374 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.488561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:41495 (local address 127.10.13.2:60431)
0925 02:31:53.488700 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:53.488716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:53.488768 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:53.488920 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:53.488920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:53.488922 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:53.488970 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.489013 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.489828 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.489829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.490447 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.490449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.490900 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.490901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.490955 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.490979 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.491215 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.491217 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:31:53.491259 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 02:31:53.491303 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260925 02:31:53.494037 18015 raft_consensus.cc:1261] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Refusing update from remote peer 8dc29e84708946a9b959218b2fa95a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:53.494037 18295 raft_consensus.cc:1261] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Refusing update from remote peer 8dc29e84708946a9b959218b2fa95a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:31:53.494318 18376 consensus_queue.cc:1035] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db62669a5b24a9fa858087dbab15ae1" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:53.494451 18438 consensus_queue.cc:1035] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da81a22a07c94a3c98232dad218f47bb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 41495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:31:53.502574 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.499771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:35003 (local address 127.10.13.3:44611)
0925 02:31:53.499840 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:53.499854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:53.499899 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:53.500089 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:53.500090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:53.500092 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:31:53.500131 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.500132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.500998 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.500998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.501619 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:53.501639 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.502153 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.502154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:53.502213 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.502220 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.502400 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.502403 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:31:53.502448 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 02:31:53.502515 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 02:31:53.502564 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.499864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:44611 (local address 127.10.13.2:35003)
0925 02:31:53.499899 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:53.499901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:53.499909 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:53.499973 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:53.499974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:53.499986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:53.500040 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:31:53.500195 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.500909 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.501735 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:53.502028 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:53.502098 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:53.502299 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:31:53.502331 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:31:53.502383 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:53.502451 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 02:31:53.502475 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:31:53.509119 17858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:53.509231 17858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:31:53.509948 17858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c54dac4515f4a75845525a447d9bc37: Sending DeleteTablet for 3 replicas of tablet 6b812bf8dd3d4a449b7b3b51402969fd
I20260925 02:31:53.510221 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b812bf8dd3d4a449b7b3b51402969fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:42630
I20260925 02:31:53.510324 17995 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b812bf8dd3d4a449b7b3b51402969fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39242
I20260925 02:31:53.510418 18392 tablet_replica.cc:330] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1: stopping tablet replica
I20260925 02:31:53.510484 18392 raft_consensus.cc:2229] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:53.510540 18392 raft_consensus.cc:2258] T 6b812bf8dd3d4a449b7b3b51402969fd P 7db62669a5b24a9fa858087dbab15ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.510618 18275 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b812bf8dd3d4a449b7b3b51402969fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:31:53 UTC) from {username='oryx', principal='oryx/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:46904
I20260925 02:31:53.510713 18394 tablet_replica.cc:330] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: stopping tablet replica
I20260925 02:31:53.510766 18394 raft_consensus.cc:2229] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:31:53.510805 18394 raft_consensus.cc:2258] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.510946 18393 tablet_replica.cc:330] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: stopping tablet replica
I20260925 02:31:53.511023 18393 raft_consensus.cc:2229] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:31:53.511085 18393 raft_consensus.cc:2258] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:31:53.511103 18394 ts_tablet_manager.cc:1905] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:53.511412 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 17930
I20260925 02:31:53.512989 18394 ts_tablet_manager.cc:1918] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:53.513052 18394 log.cc:1196] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/6b812bf8dd3d4a449b7b3b51402969fd
I20260925 02:31:53.513293 18394 ts_tablet_manager.cc:1939] T 6b812bf8dd3d4a449b7b3b51402969fd P da81a22a07c94a3c98232dad218f47bb: Deleting consensus metadata
I20260925 02:31:53.513636 17844 catalog_manager.cc:4841] TS da81a22a07c94a3c98232dad218f47bb (127.10.13.3:41495): tablet 6b812bf8dd3d4a449b7b3b51402969fd (table test-table [id=4ef6eaf3168340afb85fc648ddb3771d]) successfully deleted
I20260925 02:31:53.515466 18393 ts_tablet_manager.cc:1905] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:31:53.517119 18393 ts_tablet_manager.cc:1918] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:31:53.517189 18393 log.cc:1196] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/6b812bf8dd3d4a449b7b3b51402969fd
I20260925 02:31:53.517421 18393 ts_tablet_manager.cc:1939] T 6b812bf8dd3d4a449b7b3b51402969fd P 8dc29e84708946a9b959218b2fa95a5c: Deleting consensus metadata
I20260925 02:31:53.518095 17842 catalog_manager.cc:4841] TS 8dc29e84708946a9b959218b2fa95a5c (127.10.13.2:35003): tablet 6b812bf8dd3d4a449b7b3b51402969fd (table test-table [id=4ef6eaf3168340afb85fc648ddb3771d]) successfully deleted
W20260925 02:31:53.521272 17842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:35349 (error 108)
I20260925 02:31:53.521560 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 18070
I20260925 02:31:53.522475 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:53.521649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:35349 (local address 127.0.0.1:39254)
0925 02:31:53.522363 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:53.522410 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":39,"threads_started":1}
W20260925 02:31:53.522564 17842 catalog_manager.cc:4584] TS 7db62669a5b24a9fa858087dbab15ae1 (127.10.13.1:35349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b812bf8dd3d4a449b7b3b51402969fd: Network error: Client connection negotiation failed: client connection to 127.10.13.1:35349: connect: Connection refused (error 111)
I20260925 02:31:53.528203 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 18210
I20260925 02:31:53.534852 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 17826
2026-09-25T02:31:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2858 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:53 dist-test-slave-q8mg krb5kdc[18459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:53 dist-test-slave-q8mg krb5kdc[18459](info): set up 2 sockets
Sep 25 02:31:53 dist-test-slave-q8mg krb5kdc[18459](info): commencing operation
krb5kdc: starting...
W20260925 02:31:54.604812 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:54 dist-test-slave-q8mg krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:54.726256 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:38651
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:54.808176 18472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:54.808437 18472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:54.808473 18472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:54.809654 18472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:54.809720 18472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:54.809746 18472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:54.809769 18472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:54.809795 18472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:54.811269 18472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:38651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:38651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:54.811539 18472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:54.811760 18472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:54.816614 18477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:54.816639 18478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:54.816941 18472 server_base.cc:1034] running on GCE node
W20260925 02:31:54.816614 18480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:54.817239 18472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:54.817489 18472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:54.818627 18472 hybrid_clock.cc:648] HybridClock initialized: now 1790303514818615 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T02:31:54Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1287 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:31:54 dist-test-slave-q8mg krb5kdc[18485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:31:54 dist-test-slave-q8mg krb5kdc[18485](info): set up 2 sockets
Sep 25 02:31:54 dist-test-slave-q8mg krb5kdc[18485](info): commencing operation
krb5kdc: starting...
W20260925 02:31:55.879946 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:31:55 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:31:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:31:55.999884 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:46303
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:46303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:56.083477 18498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:56.083711 18498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:56.083747 18498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:56.084934 18498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:56.085004 18498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:56.085032 18498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:56.085060 18498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:56.085089 18498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:56.086585 18498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:46303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:46303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:56.086862 18498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:56.087092 18498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:56.092100 18504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.092149 18503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:56.092240 18498 server_base.cc:1034] running on GCE node
W20260925 02:31:56.092175 18506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:56.092543 18498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:56.092844 18498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:56.094058 18498 hybrid_clock.cc:648] HybridClock initialized: now 1790303516094037 us; error 36 us; skew 500 ppm
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:56.101285 18498 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:56.101773 18498 webserver.cc:460] Webserver started at http://127.10.13.62:36851/ using document root <none> and password file <none>
I20260925 02:31:56.102015 18498 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:56.102061 18498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:56.102157 18498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:56.103188 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "99592357d57d4cd1884d1741cac717cf"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "b2562b64c6eb114d676a26c6a11b2ba9"
server_key_iv: "0a319654d401c58abb39bdac4e1ddab5"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.103497 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "99592357d57d4cd1884d1741cac717cf"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "b2562b64c6eb114d676a26c6a11b2ba9"
server_key_iv: "0a319654d401c58abb39bdac4e1ddab5"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.104768 18498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:56.105562 18513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.105751 18498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:56.105855 18498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "99592357d57d4cd1884d1741cac717cf"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "b2562b64c6eb114d676a26c6a11b2ba9"
server_key_iv: "0a319654d401c58abb39bdac4e1ddab5"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.105926 18498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:56.118160 18498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:56.120080 18498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:56.120245 18498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:56.124792 18498 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:46303
I20260925 02:31:56.124846 18565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:46303 every 8 connection(s)
I20260925 02:31:56.125257 18498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:56.126199 18566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:31:56.128772 18566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Bootstrap starting.
I20260925 02:31:56.129426 18566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Neither blocks nor log segments found. Creating new log.
I20260925 02:31:56.129696 18566 log.cc:824] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:56.130417 18566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: No bootstrap required, opened a new log
I20260925 02:31:56.132220 18566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } }
I20260925 02:31:56.132365 18566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:31:56.132400 18566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99592357d57d4cd1884d1741cac717cf, State: Initialized, Role: FOLLOWER
I20260925 02:31:56.132555 18566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } }
I20260925 02:31:56.132642 18566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:56.132673 18566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:56.132710 18566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:31:56.133302 18566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } }
I20260925 02:31:56.133399 18566 leader_election.cc:302] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99592357d57d4cd1884d1741cac717cf; no voters:
I20260925 02:31:56.133534 18566 leader_election.cc:288] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:31:56.133627 18569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:31:56.133831 18569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 1 LEADER]: Becoming Leader. State: Replica: 99592357d57d4cd1884d1741cac717cf, State: Running, Role: LEADER
I20260925 02:31:56.133870 18566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:56.133963 18569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } }
I20260925 02:31:56.134698 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 18498
I20260925 02:31:56.134718 18570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99592357d57d4cd1884d1741cac717cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } } }
I20260925 02:31:56.134804 18570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:56.134829 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:31:56.134732 18571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99592357d57d4cd1884d1741cac717cf. Latest consensus state: current_term: 1 leader_uuid: "99592357d57d4cd1884d1741cac717cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } } }
I20260925 02:31:56.135020 18571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:56.135033 10292 external_mini_cluster.cc:1428] Setting key 987c014eecc13b674d400cec8b310183
I20260925 02:31:56.135126 18578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:56.136178 18578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:56.138041 18578 catalog_manager.cc:1347] Generated new cluster ID: 007fafcad19a4d4d8f79d5b0ffb0eb81
I20260925 02:31:56.138098 18578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:56.147259 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:56.135579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.10.13.62:46303)
0925 02:31:56.135907 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:56.135916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:56.135955 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:31:56.136083 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:56.136091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:56.136152 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:56.136296 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:56.137220 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.138777 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.139874 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.140544 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.140665 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:56.143581 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:56.143585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:56.143586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:56.143598 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:56.145619 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:56.146010 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:56.146013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:56.146014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:56.146134 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:56.146338 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:56.146339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:56.146339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:56.146661 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:56.146836 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:56.146908 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:31:56.147045 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":71,"threads_started":1}
I20260925 02:31:56.163472 18578 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:31:56.164220 18578 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:56.170542 18578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Generated new TSK 0
I20260925 02:31:56.170744 18578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:31:56.204130 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46303
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:56.286731 18592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:56.286940 18592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:56.286962 18592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:56.288067 18592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:56.288101 18592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:56.288126 18592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:56.289695 18592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:56.289885 18592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:56.290074 18592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:31:56.295100 18592 server_base.cc:1034] running on GCE node
W20260925 02:31:56.295029 18597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.295044 18598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.295083 18600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:56.295571 18592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:56.295840 18592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:56.296929 18592 hybrid_clock.cc:648] HybridClock initialized: now 1790303516296908 us; error 28 us; skew 500 ppm
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:56.299427 18592 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:31:56.299849 18592 webserver.cc:460] Webserver started at http://127.10.13.1:37945/ using document root <none> and password file <none>
I20260925 02:31:56.300086 18592 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:56.300134 18592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:56.300292 18592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:56.301379 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "174cad747fa845e48a7b693c184012c0"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "cc1a0bacfa50b320b06e0b823cb71b8f"
server_key_iv: "d39faec331aeb920e3b4dbcd8354a673"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.301734 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "174cad747fa845e48a7b693c184012c0"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "cc1a0bacfa50b320b06e0b823cb71b8f"
server_key_iv: "d39faec331aeb920e3b4dbcd8354a673"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.303048 18592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:31:56.303761 18607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.303925 18592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:56.304024 18592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "174cad747fa845e48a7b693c184012c0"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "cc1a0bacfa50b320b06e0b823cb71b8f"
server_key_iv: "d39faec331aeb920e3b4dbcd8354a673"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.304100 18592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:56.324249 18592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:56.326284 18592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:56.326462 18592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:56.326716 18592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:56.327051 18592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:56.327105 18592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.327147 18592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:56.327174 18592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.333969 18592 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:43171
I20260925 02:31:56.334025 18720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:43171 every 8 connection(s)
I20260925 02:31:56.334368 18592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:56.339259 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 18592
I20260925 02:31:56.339365 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:31:56.339586 10292 external_mini_cluster.cc:1428] Setting key e6302186d07a990a9a4421a8169d31a5
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:56.344097 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:56.335069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:51397 (local address 127.10.13.62:46303)
0925 02:31:56.335145 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:56.335148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:56.335356 (+ 208us) server_negotiation.cc:407] Connection header received
0925 02:31:56.335872 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:56.335873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:56.335887 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:56.335943 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:56.336863 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.337440 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.338399 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.338608 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.338662 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:56.341216 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:56.341219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:56.341219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:56.341228 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:56.342716 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:56.343134 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:56.343135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:56.343136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:56.343192 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:56.343441 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:56.343442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:56.343443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:56.343560 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:56.343632 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:56.343913 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 02:31:56.344005 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:31:56.344281 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:56.335065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:51397)
0925 02:31:56.335236 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:56.335275 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:56.335772 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:56.336000 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:56.336004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:56.336577 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:56.336785 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:56.336790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.337523 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.337524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:56.338324 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:56.338326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.338680 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.338681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:56.338781 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:56.339370 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:56.339381 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:56.341125 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:56.342819 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:56.342827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:56.342829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:56.343050 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:56.343258 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:56.343259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:56.343259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:56.343376 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:56.343630 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:56.343638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:56.343780 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 02:31:56.343880 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:31:56.344040 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260925 02:31:56.344782 18721 heartbeater.cc:347] Connected to a master server at 127.10.13.62:46303
I20260925 02:31:56.344935 18721 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:56.345166 18721 heartbeater.cc:511] Master 127.10.13.62:46303 requested a full tablet report, sending...
I20260925 02:31:56.345757 18530 ts_manager.cc:194] Registered new tserver with Master: 174cad747fa845e48a7b693c184012c0 (127.10.13.1:43171)
I20260925 02:31:56.346362 18530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:51397
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:31:56.367630 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46303
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:56.449292 18726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:56.449504 18726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:56.449528 18726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:56.450737 18726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:56.450780 18726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:56.450830 18726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:56.452316 18726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:56.452574 18726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:56.452809 18726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:56.457779 18732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.457803 18734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.457751 18731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:56.457985 18726 server_base.cc:1034] running on GCE node
I20260925 02:31:56.458212 18726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:56.458446 18726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:56.459600 18726 hybrid_clock.cc:648] HybridClock initialized: now 1790303516459579 us; error 37 us; skew 500 ppm
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:56.461979 18726 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:31:56.462450 18726 webserver.cc:460] Webserver started at http://127.10.13.2:45005/ using document root <none> and password file <none>
I20260925 02:31:56.462666 18726 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:56.462730 18726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:56.462890 18726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:56.463919 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "9a08d8876bf2484cb3fb9bb418e9b08f"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "bc03cc31ca92e9ead0704a9616a993f1"
server_key_iv: "b185bd158dcb641e9eb77c7bd2c9dedb"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.464272 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "9a08d8876bf2484cb3fb9bb418e9b08f"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "bc03cc31ca92e9ead0704a9616a993f1"
server_key_iv: "b185bd158dcb641e9eb77c7bd2c9dedb"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.465648 18726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:31:56.466348 18741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.466509 18726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:31:56.466606 18726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "9a08d8876bf2484cb3fb9bb418e9b08f"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "bc03cc31ca92e9ead0704a9616a993f1"
server_key_iv: "b185bd158dcb641e9eb77c7bd2c9dedb"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.466676 18726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:56.489403 18726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:56.491735 18726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:56.491894 18726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:56.492205 18726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:56.492525 18726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:56.492580 18726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.492624 18726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:56.492650 18726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.498667 18726 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:36295
I20260925 02:31:56.498710 18854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:36295 every 8 connection(s)
I20260925 02:31:56.499066 18726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:56.502611 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 18726
I20260925 02:31:56.502727 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:31:56.502909 10292 external_mini_cluster.cc:1428] Setting key 9629e61be0b8c3c0fa5a60bc3c83b9db
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:56.510155 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:56.499808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:52801 (local address 127.10.13.62:46303)
0925 02:31:56.499859 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:56.499861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:56.500078 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:31:56.500676 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:56.500677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:56.500691 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:56.500765 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:56.501976 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.502748 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.503958 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.504174 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.504260 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:56.506966 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:56.506967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:56.506967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:56.506974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:56.508512 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:56.509139 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:56.509140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:56.509141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:56.509206 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:56.509506 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:56.509507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:56.509507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:56.509626 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:56.509676 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:56.510020 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 02:31:56.510091 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:31:56.510422 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:56.499776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:52801)
0925 02:31:56.499932 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:56.499975 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:56.500553 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:56.500858 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:56.500863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:56.501574 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:56.501855 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:56.501861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.502836 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.502837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:56.503877 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:56.503880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.504273 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.504274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:56.504404 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:56.504975 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:56.504986 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:56.506861 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:56.508627 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:56.508635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:56.508638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:56.509013 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:56.509273 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:56.509274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:56.509274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:56.509433 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:56.509688 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:56.509695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:56.509877 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 02:31:56.509989 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:31:56.510166 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260925 02:31:56.510998 18855 heartbeater.cc:347] Connected to a master server at 127.10.13.62:46303
I20260925 02:31:56.511109 18855 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:56.511327 18855 heartbeater.cc:511] Master 127.10.13.62:46303 requested a full tablet report, sending...
I20260925 02:31:56.511737 18530 ts_manager.cc:194] Registered new tserver with Master: 9a08d8876bf2484cb3fb9bb418e9b08f (127.10.13.2:36295)
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
I20260925 02:31:56.512127 18530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:52801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:31:56.529716 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46303
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:56.612798 18860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:56.613116 18860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:56.613158 18860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:56.614362 18860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:56.614425 18860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:56.614470 18860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:56.615940 18860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:56.616230 18860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:56.616464 18860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:56.621389 18866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:56.621599 18860 server_base.cc:1034] running on GCE node
W20260925 02:31:56.621367 18868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.621371 18865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:56.621949 18860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:56.622162 18860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:56.623291 18860 hybrid_clock.cc:648] HybridClock initialized: now 1790303516623282 us; error 28 us; skew 500 ppm
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:56.625738 18860 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:31:56.626212 18860 webserver.cc:460] Webserver started at http://127.10.13.3:32935/ using document root <none> and password file <none>
I20260925 02:31:56.626412 18860 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:56.626452 18860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:56.626541 18860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:31:56.627532 18860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "e7b8e3595bdb4e739ed4e928582817c9"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "a2eb7b7a82b283ce7373134081a1e830"
server_key_iv: "488589dc822f914de2183ccac5d09cac"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.627833 18860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "e7b8e3595bdb4e739ed4e928582817c9"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "a2eb7b7a82b283ce7373134081a1e830"
server_key_iv: "488589dc822f914de2183ccac5d09cac"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.629168 18860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:31:56.629823 18875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.630075 18860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:31:56.630198 18860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "e7b8e3595bdb4e739ed4e928582817c9"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "a2eb7b7a82b283ce7373134081a1e830"
server_key_iv: "488589dc822f914de2183ccac5d09cac"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.630275 18860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:56.648164 18860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:56.650123 18860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:56.650242 18860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:56.650432 18860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:56.650705 18860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:56.650736 18860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.650758 18860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:56.650770 18860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.656514 18860 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:34039
I20260925 02:31:56.656590 18988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:34039 every 8 connection(s)
I20260925 02:31:56.656944 18860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:31:56.664352 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 18860
I20260925 02:31:56.664458 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:31:56.664649 10292 external_mini_cluster.cc:1428] Setting key 88c15150a898a9e45959396aab8bc21a
I20260925 02:31:56.666329 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:56.657693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:38499 (local address 127.10.13.62:46303)
0925 02:31:56.657741 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:56.657743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:56.657944 (+ 201us) server_negotiation.cc:407] Connection header received
0925 02:31:56.658474 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:56.658475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:56.658490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:56.658541 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:56.659473 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.659986 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.661014 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.661221 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.661303 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:56.663428 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:56.663429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:56.663430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:56.663437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:56.664971 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:56.665484 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:56.665485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:56.665486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:56.665546 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:56.665791 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:56.665792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:56.665792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:56.665894 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:56.665947 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:56.666206 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:31:56.666267 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:31:56.666545 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:56.657667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:38499)
0925 02:31:56.657833 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:56.657867 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:56.658366 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:56.658608 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:56.658612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:56.659183 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:56.659377 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:56.659382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.660085 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.660086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:56.660925 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:56.660928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:56.661314 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:56.661315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:56.661446 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:56.661955 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:56.661964 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:56.663342 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:31:56.665068 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:56.665075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:56.665078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:56.665390 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:56.665628 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:31:56.665629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:31:56.665629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:31:56.665737 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:31:56.665951 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:31:56.665955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:31:56.666095 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 02:31:56.666180 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:31:56.666298 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260925 02:31:56.667006 18989 heartbeater.cc:347] Connected to a master server at 127.10.13.62:46303
I20260925 02:31:56.667088 18989 heartbeater.cc:464] Registering TS with master...
I20260925 02:31:56.667276 18989 heartbeater.cc:511] Master 127.10.13.62:46303 requested a full tablet report, sending...
I20260925 02:31:56.667631 18530 ts_manager.cc:194] Registered new tserver with Master: e7b8e3595bdb4e739ed4e928582817c9 (127.10.13.3:34039)
I20260925 02:31:56.667990 18530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:38499
I20260925 02:31:56.676513 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab.
Entry for principal oryx/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab.
WARNING: no policy specified for oryx/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab.
Entry for principal oryx/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab.
WARNING: no policy specified for oryx/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab.
Entry for principal oryx/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab.
I20260925 02:31:56.747418 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 18592
I20260925 02:31:56.752943 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 18726
I20260925 02:31:56.757550 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 18860
I20260925 02:31:56.761834 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 18498
I20260925 02:31:56.767010 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:46303
--webserver_interface=127.10.13.62
--webserver_port=36851
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:46303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:31:56.848510 18998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:56.848713 18998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:56.848737 18998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:56.849900 18998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:31:56.849941 18998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:56.849954 18998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:56.849967 18998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:31:56.849979 18998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:31:56.851372 18998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:46303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:46303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=36851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:56.851634 18998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:56.851862 18998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:56.856609 19004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.856622 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.856647 19003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:56.856930 18998 server_base.cc:1034] running on GCE node
I20260925 02:31:56.857173 18998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:56.857410 18998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:56.858604 18998 hybrid_clock.cc:648] HybridClock initialized: now 1790303516858586 us; error 27 us; skew 500 ppm
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:56.861112 18998 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:31:56.861605 18998 webserver.cc:460] Webserver started at http://127.10.13.62:36851/ using document root <none> and password file <none>
I20260925 02:31:56.861825 18998 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:56.861903 18998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:56.863240 18998 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:56.864101 19013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:56.864274 18998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:56.864380 18998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "99592357d57d4cd1884d1741cac717cf"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "b2562b64c6eb114d676a26c6a11b2ba9"
server_key_iv: "0a319654d401c58abb39bdac4e1ddab5"
server_key_version: "encryptionkey@0"
I20260925 02:31:56.864722 18998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:56.893160 18998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:56.895576 18998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:56.895742 18998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:56.899890 18998 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:46303
I20260925 02:31:56.899930 19065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:46303 every 8 connection(s)
I20260925 02:31:56.900352 18998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:31:56.901868 19066 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:31:56.902202 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 18998
I20260925 02:31:56.902482 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:43171
--local_ip_for_outbound_sockets=127.10.13.1
--tserver_master_addrs=127.10.13.62:46303
--webserver_port=37945
--webserver_interface=127.10.13.1
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.13.1
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:31:56.902649 19066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Bootstrap starting.
I20260925 02:31:56.911144 19066 log.cc:824] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:31:56.913081 19066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:31:56.913365 19066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Bootstrap complete.
I20260925 02:31:56.915650 19066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } }
I20260925 02:31:56.915926 19066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99592357d57d4cd1884d1741cac717cf, State: Initialized, Role: FOLLOWER
I20260925 02:31:56.916113 19066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } }
I20260925 02:31:56.916172 19066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:31:56.916230 19066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:31:56.916276 19066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:31:56.917320 19066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } }
I20260925 02:31:56.917403 19066 leader_election.cc:302] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99592357d57d4cd1884d1741cac717cf; no voters:
I20260925 02:31:56.917608 19066 leader_election.cc:288] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:31:56.917644 19070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:31:56.917764 19070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [term 2 LEADER]: Becoming Leader. State: Replica: 99592357d57d4cd1884d1741cac717cf, State: Running, Role: LEADER
I20260925 02:31:56.917892 19066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:31:56.917923 19070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } }
I20260925 02:31:56.918210 19070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99592357d57d4cd1884d1741cac717cf. Latest consensus state: current_term: 2 leader_uuid: "99592357d57d4cd1884d1741cac717cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } } }
I20260925 02:31:56.918285 19070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:56.918212 19072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "99592357d57d4cd1884d1741cac717cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99592357d57d4cd1884d1741cac717cf" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 46303 } } }
I20260925 02:31:56.918408 19072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf [sys.catalog]: This master's current role is: LEADER
I20260925 02:31:56.918521 19075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:31:56.919039 19075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:31:56.919203 19075 catalog_manager.cc:1259] Loaded cluster ID: 007fafcad19a4d4d8f79d5b0ffb0eb81
I20260925 02:31:56.919410 19075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:31:56.920583 19075 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:31:56.920749 19075 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 99592357d57d4cd1884d1741cac717cf: Loaded TSK: 0
I20260925 02:31:56.920966 19075 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:31:56.985795 19068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:56.986002 19068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:56.986027 19068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:56.987180 19068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:56.987221 19068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:56.987246 19068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:31:56.988646 19068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.1.keytab
--principal=oryx/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:43171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=37945
--tserver_master_addrs=127.10.13.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:56.988873 19068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:56.989135 19068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:31:56.994225 19068 server_base.cc:1034] running on GCE node
W20260925 02:31:56.994194 19093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.994297 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:56.994177 19092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:56.994632 19068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:56.994894 19068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:56.996073 19068 hybrid_clock.cc:648] HybridClock initialized: now 1790303516996049 us; error 38 us; skew 500 ppm
Sep 25 02:31:56 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303516, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:56.998641 19068 init.cc:377] Logged in from keytab as oryx/127.10.13.1@KRBTEST.COM (short username oryx)
I20260925 02:31:56.999109 19068 webserver.cc:460] Webserver started at http://127.10.13.1:37945/ using document root <none> and password file <none>
I20260925 02:31:56.999330 19068 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:56.999411 19068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:57.000823 19068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:57.001551 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.001722 19068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:31:57.001825 19068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "174cad747fa845e48a7b693c184012c0"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "cc1a0bacfa50b320b06e0b823cb71b8f"
server_key_iv: "d39faec331aeb920e3b4dbcd8354a673"
server_key_version: "encryptionkey@0"
I20260925 02:31:57.002166 19068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:57.025116 19068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:57.027384 19068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:57.027556 19068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:57.027812 19068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:57.028163 19068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:57.028216 19068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.028258 19068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:57.028285 19068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.034893 19068 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:43171
I20260925 02:31:57.034933 19215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:43171 every 8 connection(s)
I20260925 02:31:57.035298 19068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:31:57.037739 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 19068
I20260925 02:31:57.038050 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:36295
--local_ip_for_outbound_sockets=127.10.13.2
--tserver_master_addrs=127.10.13.62:46303
--webserver_port=45005
--webserver_interface=127.10.13.2
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.13.2
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:57.046276 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.036075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:43343)
0925 02:31:57.036267 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.036317 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.036792 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.037392 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.037396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.038137 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.038408 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.038414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.039739 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.039740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.041045 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.041049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.042076 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.042077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.042227 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.042893 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.042907 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.046021 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
W20260925 02:31:57.046729 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.036160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:43343 (local address 127.10.13.62:46303)
0925 02:31:57.037005 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.037019 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.037049 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:31:57.037159 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.037164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.037205 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.037340 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.038524 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.039625 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.041135 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.041821 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.041955 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.046177 (+ 4222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:43343: BlockingRecv error: recv got EOF from 127.10.13.1:43343 (error 108)
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":88,"threads_started":1}
W20260925 02:31:57.046751 19216 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:46303 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.13.62:46303: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:57.052810 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.047059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:60669)
0925 02:31:57.047176 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.047186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.047224 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.047399 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.047400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.047559 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.047618 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.047620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.048633 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.048635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.049567 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.049570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.050093 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.050094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.050166 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.050662 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.050668 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.052737 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260925 02:31:57.053138 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.047157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:60669 (local address 127.10.13.62:46303)
0925 02:31:57.047200 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.047202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.047215 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:31:57.047296 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.047297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.047312 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.047390 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.047703 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.048532 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.049675 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.050005 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.050080 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.052837 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:60669: BlockingRecv error: recv got EOF from 127.10.13.1:60669 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:57.057838 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.053326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:52287)
0925 02:31:57.053447 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.053456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.053534 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.053685 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.053685 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.053842 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.053871 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.053873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.054530 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.054531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.055083 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.055085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.055448 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.055450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.055514 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.055967 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.055972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.057765 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260925 02:31:57.057909 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.053445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:52287 (local address 127.10.13.62:46303)
0925 02:31:57.053513 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.053515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.053525 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:57.053625 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.053625 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.053636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.053683 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.053958 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.054436 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.055170 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.055363 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.055440 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.057844 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:52287: BlockingRecv error: recv got EOF from 127.10.13.1:52287 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 02:31:57.057969 19216 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:31:57.126626 19220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:57.126821 19220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:57.126845 19220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:57.127954 19220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:57.127986 19220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:57.128018 19220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:31:57.129669 19220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.2.keytab
--principal=oryx/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:36295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=45005
--tserver_master_addrs=127.10.13.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:57.129853 19220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:57.130031 19220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:57.135141 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:57.135197 19228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:57.135310 19220 server_base.cc:1034] running on GCE node
W20260925 02:31:57.135226 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:57.135542 19220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:57.135753 19220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:57.136976 19220 hybrid_clock.cc:648] HybridClock initialized: now 1790303517136954 us; error 34 us; skew 500 ppm
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303517, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:57.139485 19220 init.cc:377] Logged in from keytab as oryx/127.10.13.2@KRBTEST.COM (short username oryx)
I20260925 02:31:57.139901 19220 webserver.cc:460] Webserver started at http://127.10.13.2:45005/ using document root <none> and password file <none>
I20260925 02:31:57.140146 19220 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:57.140228 19220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:57.141660 19220 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:57.142282 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.142464 19220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:57.142555 19220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "9a08d8876bf2484cb3fb9bb418e9b08f"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "bc03cc31ca92e9ead0704a9616a993f1"
server_key_iv: "b185bd158dcb641e9eb77c7bd2c9dedb"
server_key_version: "encryptionkey@0"
I20260925 02:31:57.142897 19220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:57.173187 19220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:57.175483 19220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:57.175648 19220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:57.175904 19220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:57.176223 19220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:57.176277 19220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.176321 19220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:57.176366 19220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.182256 19220 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:36295
I20260925 02:31:57.182304 19348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:36295 every 8 connection(s)
I20260925 02:31:57.182652 19220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:31:57.183784 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 19220
I20260925 02:31:57.184091 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:34039
--local_ip_for_outbound_sockets=127.10.13.3
--tserver_master_addrs=127.10.13.62:46303
--webserver_port=32935
--webserver_interface=127.10.13.3
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.13.3
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:31:57.191959 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.183356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:47317 (local address 127.10.13.62:46303)
0925 02:31:57.183424 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.183426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.183637 (+ 211us) server_negotiation.cc:407] Connection header received
0925 02:31:57.184405 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.184406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.184421 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.184480 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.185844 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.186657 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.188225 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.188431 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.188487 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.191890 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:47317: BlockingRecv error: recv got EOF from 127.10.13.2:47317 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:31:57.192036 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.183346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:47317)
0925 02:31:57.183502 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.183540 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.184292 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.184578 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.184582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.185441 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.185746 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.185752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.186753 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.186755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.188118 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.188134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.188518 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.188520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.188680 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.189249 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.189261 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.191812 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
W20260925 02:31:57.192248 19349 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:46303 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.13.62:46303: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:57.196674 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.192473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:45753)
0925 02:31:57.192576 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.192588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.192636 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.192765 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.192766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.192942 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.192992 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.192994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.193611 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.193612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.194130 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.194132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.194479 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.194480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.194547 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.194966 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.194972 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.196601 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260925 02:31:57.196774 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.192537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:45753 (local address 127.10.13.62:46303)
0925 02:31:57.192579 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.192581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.192634 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:31:57.192701 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.192702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.192711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.192756 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.193072 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.193535 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.194211 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.194408 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.194479 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.196688 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:45753: BlockingRecv error: recv got EOF from 127.10.13.2:45753 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:57.201023 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.197112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:35701)
0925 02:31:57.197158 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.197171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.197211 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.197321 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.197322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.197439 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.197470 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.197472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.198061 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.198062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.198571 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.198573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.198926 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.198927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.198981 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.199366 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.199371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.200941 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 02:31:57.201128 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.197187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:35701 (local address 127.10.13.62:46303)
0925 02:31:57.197223 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.197225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.197233 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:31:57.197264 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.197264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.197273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.197320 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.197538 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.197985 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.198650 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.198836 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.198924 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.201018 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:35701: BlockingRecv error: recv got EOF from 127.10.13.2:35701 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 02:31:57.201151 19349 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:31:57.271001 19352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:31:57.271224 19352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:31:57.271263 19352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:31:57.272444 19352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:31:57.272504 19352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:31:57.272549 19352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:31:57.274070 19352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/krb5kdc/oryx_127.10.13.3.keytab
--principal=oryx/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:34039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=32935
--tserver_master_addrs=127.10.13.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:31:57.274324 19352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:31:57.274546 19352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:31:57.279536 19358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:57.279560 19360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:31:57.279536 19357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:31:57.279799 19352 server_base.cc:1034] running on GCE node
I20260925 02:31:57.280053 19352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:31:57.280293 19352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:31:57.281458 19352 hybrid_clock.cc:648] HybridClock initialized: now 1790303517281448 us; error 36 us; skew 500 ppm
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303517, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:31:57.283931 19352 init.cc:377] Logged in from keytab as oryx/127.10.13.3@KRBTEST.COM (short username oryx)
I20260925 02:31:57.284366 19352 webserver.cc:460] Webserver started at http://127.10.13.3:32935/ using document root <none> and password file <none>
I20260925 02:31:57.284615 19352 fs_manager.cc:359] Metadata directory not provided
I20260925 02:31:57.284700 19352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:31:57.286083 19352 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:31:57.286649 19367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.286808 19352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:31:57.286895 19352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "e7b8e3595bdb4e739ed4e928582817c9"
format_stamp: "Formatted at 2026-09-25 02:31:56 on dist-test-slave-q8mg"
server_key: "a2eb7b7a82b283ce7373134081a1e830"
server_key_iv: "488589dc822f914de2183ccac5d09cac"
server_key_version: "encryptionkey@0"
I20260925 02:31:57.287240 19352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:31:57.331791 19352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:31:57.334172 19352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:31:57.334340 19352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:31:57.334600 19352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:31:57.334913 19352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:31:57.334964 19352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.335005 19352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:31:57.335034 19352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:31:57.340968 19352 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:34039
I20260925 02:31:57.341017 19480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:34039 every 8 connection(s)
I20260925 02:31:57.341378 19352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:31:57.349031 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.342054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:60891 (local address 127.10.13.62:46303)
0925 02:31:57.342137 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.342140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.342345 (+ 205us) server_negotiation.cc:407] Connection header received
0925 02:31:57.342904 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.342905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.342916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.342997 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.343912 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.344485 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.345528 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.345823 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.345896 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.348973 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:60891: BlockingRecv error: recv got EOF from 127.10.13.3:60891 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:31:57.349098 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.342029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:60891)
0925 02:31:57.342203 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.342241 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.342812 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.343051 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.343054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.343632 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.343828 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.343832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.344562 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.344563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.345417 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.345419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.345944 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.345945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.346078 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.346591 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.346601 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.348868 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
W20260925 02:31:57.349352 19481 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:46303 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.13.62:46303: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
I20260925 02:31:57.351223 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 19352
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:57.354404 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.349617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:55583)
0925 02:31:57.349677 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.349688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.349730 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.349993 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.349994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.350133 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.350184 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.350185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.350804 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.350806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.351508 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.351509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.351956 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.351957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.352038 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.352435 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.352441 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.354315 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:31:57.354453 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.349834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:55583 (local address 127.10.13.62:46303)
0925 02:31:57.349889 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.349891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.349897 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:31:57.349917 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.349918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.349927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.349989 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.350259 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.350729 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.351589 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.351862 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.351932 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.354388 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:55583: BlockingRecv error: recv got EOF from 127.10.13.3:55583 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:57 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:57.359172 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.351610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.10.13.62:46303)
0925 02:31:57.351743 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.351745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.351845 (+ 100us) server_negotiation.cc:407] Connection header received
0925 02:31:57.351891 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.351891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.351904 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.351967 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.352714 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.353238 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.354062 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.354273 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.354352 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.355836 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:31:57.355841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:31:57.355842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:31:57.355853 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:31:57.357782 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:57.358135 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:57.358139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:57.358139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:57.358242 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:31:57.358434 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:31:57.358434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:31:57.358435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:31:57.358691 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:31:57.358844 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:31:57.358961 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 02:31:57.359079 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260925 02:31:57.359318 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:57.354721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:46465)
0925 02:31:57.354926 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:57.354935 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:57.354975 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:57.355157 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:57.355158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:57.355264 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:57.355291 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.355292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.356081 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.356082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.356859 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:57.356862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.357204 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.357205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:57.357256 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.357543 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:57.357549 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:57.359239 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190}
W20260925 02:31:57.359354 19219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:57.354778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:46465 (local address 127.10.13.62:46303)
0925 02:31:57.354840 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:57.354842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:57.354982 (+ 140us) server_negotiation.cc:407] Connection header received
0925 02:31:57.355044 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:57.355045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:57.355065 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:57.355143 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:57.355356 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.356010 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.356964 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:57.357138 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:57.357187 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:57.359296 (+ 2109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:46465: BlockingRecv error: recv got EOF from 127.10.13.3:46465 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 02:31:57.359452 19481 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 02:31:58 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:58 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:58.062934 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:58.058398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:59191)
0925 02:31:58.058535 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:58.058545 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:58.058610 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:58.058790 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:58.058791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:58.058960 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:58.059039 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:58.059041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.059684 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.059685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:58.060239 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:58.060241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.060631 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.060632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:58.060671 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:58.061174 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:58.061179 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:58.062865 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
W20260925 02:31:58.063118 19487 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:58.058456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:59191 (local address 127.10.13.62:46303)
0925 02:31:58.058592 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:58.058594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:58.058605 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:58.058684 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:58.058684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:58.058698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:58.058772 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:58.059117 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.059620 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.060332 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.060558 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.060622 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:58.063006 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:59191: BlockingRecv error: recv got EOF from 127.10.13.1:59191 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
Sep 25 02:31:58 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:58 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:58.206211 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:58.201593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:54215)
0925 02:31:58.201736 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:58.201746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:58.201802 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:58.202037 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:58.202039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:58.202200 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:58.202251 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:58.202253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.202914 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.202915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:58.203458 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:58.203460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.203815 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.203816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:58.203856 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:58.204337 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:58.204342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:58.206132 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260925 02:31:58.206358 19487 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:58.201670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:54215 (local address 127.10.13.62:46303)
0925 02:31:58.201748 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:58.201750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:58.201790 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:31:58.201882 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:58.201882 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:58.201896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:58.201990 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:58.202329 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.202847 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.203543 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.203748 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.203805 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:58.206273 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:54215: BlockingRecv error: recv got EOF from 127.10.13.2:54215 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:31:58 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:58 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:58.364578 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:58.359869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:49237)
0925 02:31:58.359992 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:58.360005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:58.360066 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:58.360221 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:58.360222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:58.360395 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:58.360445 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:58.360447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.361169 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.361169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:58.361706 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:58.361707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.362218 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.362219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:58.362286 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:58.362752 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:58.362757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:58.364507 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
W20260925 02:31:58.364665 19487 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:58.359949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:49237 (local address 127.10.13.62:46303)
0925 02:31:58.360042 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:58.360044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:58.360054 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:31:58.360136 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:58.360137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:58.360151 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:58.360203 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:58.360534 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.361103 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.361788 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:58.362121 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:58.362181 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:58.364589 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:49237: BlockingRecv error: recv got EOF from 127.10.13.3:49237 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 02:31:59 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:59 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:59.068512 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:59.063374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:41733)
0925 02:31:59.063522 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:59.063534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:59.063589 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:59.063742 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:59.063742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:59.063908 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:59.063965 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:59.063966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.064679 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.064680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:59.065304 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:59.065308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.065724 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.065725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:59.065773 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:59.066344 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:59.066349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:59.068402 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
W20260925 02:31:59.068567 19491 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:59.063463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:41733 (local address 127.10.13.62:46303)
0925 02:31:59.063615 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:59.063616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:59.063630 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:31:59.063653 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:59.063654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:59.063665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:59.063721 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:59.064041 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.064586 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.065394 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.065647 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.065702 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:59.068490 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:41733: BlockingRecv error: recv got EOF from 127.10.13.1:41733 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
Sep 25 02:31:59 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:59 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:59.217669 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:59.206680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:57219)
0925 02:31:59.206819 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:59.206835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:59.206898 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:59.207156 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:59.207157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:59.207323 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:59.207381 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:59.207382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.208022 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.208024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:59.208551 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:59.208553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.208908 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.208909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:59.208971 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:59.215344 (+ 6373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:59.215351 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:59.217585 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
W20260925 02:31:59.217744 19491 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:59.206724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:57219 (local address 127.10.13.62:46303)
0925 02:31:59.206793 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:59.206796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:59.206883 (+ 87us) server_negotiation.cc:407] Connection header received
0925 02:31:59.207017 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:59.207018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:59.207032 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:59.207109 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:59.207445 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.207930 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.208631 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.208804 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.208853 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:59.217692 (+ 8839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:57219: BlockingRecv error: recv got EOF from 127.10.13.2:57219 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 02:31:59 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:31:59 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:31:59.370028 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:31:59.365250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:60115)
0925 02:31:59.365436 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 02:31:59.365449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:31:59.365507 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:31:59.365668 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:31:59.365669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:31:59.365823 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:31:59.365874 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:59.365875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.366514 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.366514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:59.367033 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:31:59.367034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.367451 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.367452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:31:59.367490 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:59.367981 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:31:59.367985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:31:59.369965 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":58,"threads_started":1}
W20260925 02:31:59.370087 19491 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:31:59.365443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:60115 (local address 127.10.13.62:46303)
0925 02:31:59.365513 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:31:59.365515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:31:59.365526 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:31:59.365565 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:31:59.365566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:31:59.365580 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:31:59.365645 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:31:59.365937 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.366439 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.367174 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:31:59.367384 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:31:59.367439 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:31:59.370039 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:60115: BlockingRecv error: recv got EOF from 127.10.13.3:60115 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 02:32:00 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:00 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:00.073917 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:00.069088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:59823)
0925 02:32:00.069253 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:00.069264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:00.069326 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:00.069492 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:00.069493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:00.069671 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:00.069723 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:00.069724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.070637 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.070639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:00.071254 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:00.071257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.071605 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.071606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:00.071657 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:00.072077 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:00.072082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:00.073836 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
W20260925 02:32:00.074054 19495 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:00.069174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:59823 (local address 127.10.13.62:46303)
0925 02:32:00.069299 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:00.069301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:00.069312 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:32:00.069380 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:00.069381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:00.069393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:00.069476 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:00.069824 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.070533 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.071333 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.071537 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.071593 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:00.073950 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:59823: BlockingRecv error: recv got EOF from 127.10.13.1:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
Sep 25 02:32:00 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:00 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:00.223104 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:00.218233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:41419)
0925 02:32:00.218379 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:00.218388 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:00.218446 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:00.218590 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:00.218591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:00.218871 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:00.218929 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:00.218932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.219649 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.219650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:00.220175 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:00.220176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.220549 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.220550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:00.220588 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:00.221109 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:00.221114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:00.222968 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260925 02:32:00.223109 19495 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:00.218293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:41419 (local address 127.10.13.62:46303)
0925 02:32:00.218378 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:00.218380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:00.218453 (+ 73us) server_negotiation.cc:407] Connection header received
0925 02:32:00.218501 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:00.218502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:00.218516 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:00.218577 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:00.219008 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.219568 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.220248 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.220450 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.220540 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:00.223060 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:41419: BlockingRecv error: recv got EOF from 127.10.13.2:41419 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 02:32:00 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:00 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:00.375329 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:00.370570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:58491)
0925 02:32:00.370736 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:00.370748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:00.370809 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:00.370957 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:00.370958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:00.371116 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:00.371184 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:00.371185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.371816 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.371816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:00.372366 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:00.372367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.372718 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.372719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:00.372759 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:00.373316 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:00.373321 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:00.375255 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
W20260925 02:32:00.375509 19495 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:00.370662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:58491 (local address 127.10.13.62:46303)
0925 02:32:00.370752 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:00.370754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:00.370798 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:32:00.370866 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:00.370867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:00.370881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:00.370939 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:00.371252 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.371747 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.372453 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:00.372646 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:00.372702 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:00.375403 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:58491: BlockingRecv error: recv got EOF from 127.10.13.3:58491 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 02:32:01 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:01 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:01.079540 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:01.074493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:48777)
0925 02:32:01.074681 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:01.074696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:01.074777 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:01.074999 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:01.075001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:01.075178 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:01.075237 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:01.075239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.075927 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.075928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:01.076563 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:01.076565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.076951 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.076953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:01.077063 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:01.077537 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:01.077542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:01.079457 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
W20260925 02:32:01.079612 19499 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:01.074617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:48777 (local address 127.10.13.62:46303)
0925 02:32:01.074790 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:01.074792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:01.074800 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:32:01.074829 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:01.074829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:01.074840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:01.074903 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:01.075324 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.075832 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.076650 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.076840 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.076917 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:01.079556 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:48777: BlockingRecv error: recv got EOF from 127.10.13.1:48777 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
Sep 25 02:32:01 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:01 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:01.234091 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:01.223674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:34425)
0925 02:32:01.223812 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:01.223826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:01.223898 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:01.224091 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:01.224092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:01.224294 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:01.224346 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:01.224348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.225098 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.225099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:01.225646 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:01.225648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.226015 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.226016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:01.226076 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:01.232124 (+ 6048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:01.232131 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:01.233991 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
W20260925 02:32:01.234156 19499 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:01.223740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:34425 (local address 127.10.13.62:46303)
0925 02:32:01.223790 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:01.223792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:01.223905 (+ 113us) server_negotiation.cc:407] Connection header received
0925 02:32:01.223993 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:01.223994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:01.224007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:01.224077 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:01.224447 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.225016 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.225736 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.225938 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.226016 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:01.234095 (+ 8079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:34425: BlockingRecv error: recv got EOF from 127.10.13.2:34425 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 02:32:01 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:01 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:01.380950 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:01.375965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:36289)
0925 02:32:01.376134 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:01.376145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:01.376229 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:01.376398 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:01.376399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:01.376577 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:01.376629 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:01.376631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.377400 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.377403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:01.377953 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:01.377954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.378414 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.378416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:01.378481 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:01.379016 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:01.379022 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:01.380825 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
W20260925 02:32:01.381003 19499 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:01.376144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:36289 (local address 127.10.13.62:46303)
0925 02:32:01.376221 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:01.376223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:01.376235 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:32:01.376300 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:01.376301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:01.376315 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:01.376388 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:01.376716 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.377305 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.378039 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:01.378338 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:01.378408 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:01.380935 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:36289: BlockingRecv error: recv got EOF from 127.10.13.3:36289 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 02:32:02 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:02 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:02.085260 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:02.080158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:33277)
0925 02:32:02.080314 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:02.080325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:02.080381 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:02.080641 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:02.080643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:02.080876 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:02.080965 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:02.080968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.081669 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.081670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:02.082203 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:02.082205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.082580 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.082581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:02.082646 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:02.083212 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:02.083218 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:02.085158 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
W20260925 02:32:02.085314 19503 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:02.080304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:33277 (local address 127.10.13.62:46303)
0925 02:32:02.080443 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:02.080445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:02.080453 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:32:02.080482 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:02.080483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:02.080496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:02.080575 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:02.081065 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.081580 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.082288 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.082501 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.082551 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:02.085239 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:33277: BlockingRecv error: recv got EOF from 127.10.13.1:33277 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
Sep 25 02:32:02 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:02 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:02.239660 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:02.234709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:53215)
0925 02:32:02.234855 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:02.234870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:02.234939 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:02.235142 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:02.235143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:02.235325 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:02.235382 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:02.235384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.236032 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.236033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:02.236678 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:02.236681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.237233 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.237234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:02.237304 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:02.237751 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:02.237757 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:02.239591 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
W20260925 02:32:02.239809 19503 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:02.234829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:53215 (local address 127.10.13.62:46303)
0925 02:32:02.234919 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:02.234921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:02.234932 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:32:02.235049 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:02.235050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:02.235062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:02.235120 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:02.235446 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.235936 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.236771 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.237129 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.237203 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:02.239723 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:53215: BlockingRecv error: recv got EOF from 127.10.13.2:53215 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:32:02 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:02 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:02.386150 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:02.381511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:35741)
0925 02:32:02.381664 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:02.381680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:02.381738 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:02.381882 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:02.381883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:02.382096 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:02.382148 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:02.382150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.382821 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.382822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:02.383403 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:02.383405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.383742 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.383743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:02.383793 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:02.384268 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:02.384273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:02.386085 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
W20260925 02:32:02.386269 19503 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:02.381592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:35741 (local address 127.10.13.62:46303)
0925 02:32:02.381680 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:02.381682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:02.381729 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:32:02.381803 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:02.381804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:02.381816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:02.381858 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:02.382231 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.382740 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.383479 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:02.383658 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:02.383709 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:02.386190 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:35741: BlockingRecv error: recv got EOF from 127.10.13.3:35741 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 02:32:03 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:03 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:03.090499 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:03.085750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:51609)
0925 02:32:03.085877 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:03.085892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:03.085982 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:03.086218 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:03.086219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:03.086376 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:03.086430 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:03.086432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.087098 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.087098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:03.087628 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:03.087629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.087976 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.087977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:03.088018 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:03.088550 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:03.088555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:03.090428 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
W20260925 02:32:03.090631 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:03.085844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:51609 (local address 127.10.13.62:46303)
0925 02:32:03.086085 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:03.086087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:03.086097 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:32:03.086121 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:03.086122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:03.086133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:03.086201 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:03.086502 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.087005 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.087706 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.087900 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.087968 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:03.090507 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:51609: BlockingRecv error: recv got EOF from 127.10.13.1:51609 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
Sep 25 02:32:03 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:03 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:03.244831 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:03.240156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:40967)
0925 02:32:03.240306 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:03.240316 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:03.240373 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:03.240531 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:03.240531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:03.240702 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:03.240769 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:03.240771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.241423 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.241424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:03.241959 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:03.241961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.242338 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.242339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:03.242378 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:03.242845 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:03.242850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:03.244769 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20260925 02:32:03.245136 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:03.240259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:40967 (local address 127.10.13.62:46303)
0925 02:32:03.240321 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:03.240323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:03.240367 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:32:03.240452 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:03.240453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:03.240464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:03.240510 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:03.240838 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.241356 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.242059 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.242264 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.242329 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:03.244910 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:40967: BlockingRecv error: recv got EOF from 127.10.13.2:40967 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 02:32:03 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:03 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:03.392067 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:03.386758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:58471)
0925 02:32:03.386973 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:03.386984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:03.387055 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:03.387223 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:03.387223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:03.387392 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:03.387442 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:03.387443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.388146 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.388147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:03.388812 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:03.388815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.389416 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.389418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:03.389485 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:03.389943 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:03.389948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:03.391972 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
W20260925 02:32:03.392143 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:03.386885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:58471 (local address 127.10.13.62:46303)
0925 02:32:03.386939 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:03.386941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:03.387083 (+ 142us) server_negotiation.cc:407] Connection header received
0925 02:32:03.387120 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:03.387121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:03.387138 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:03.387203 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:03.387514 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.388051 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.389049 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:03.389336 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:03.389398 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:03.392074 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:58471: BlockingRecv error: recv got EOF from 127.10.13.3:58471 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 02:32:04 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:04 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:04.095979 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:04.090979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:50925)
0925 02:32:04.091156 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:04.091169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:04.091230 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:04.091393 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:04.091394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:04.091611 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:04.091664 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:04.091666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.092344 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.092346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:04.092908 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:04.092909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.093280 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.093281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:04.093342 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:04.093833 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:04.093838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:04.095873 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
W20260925 02:32:04.096031 19511 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:04.091042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:50925 (local address 127.10.13.62:46303)
0925 02:32:04.091210 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:04.091212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:04.091221 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:32:04.091280 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:04.091281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:04.091292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:04.091379 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:04.091749 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.092260 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.092998 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.093204 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.093281 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:04.095968 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:50925: BlockingRecv error: recv got EOF from 127.10.13.1:50925 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
Sep 25 02:32:04 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:04 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:04.249933 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:04.245436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:45591)
0925 02:32:04.245569 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:04.245582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:04.245647 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:04.245909 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:04.245909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:04.246080 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:04.246134 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:04.246136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.246785 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.246786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:04.247338 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:04.247339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.247671 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.247672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:04.247710 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:04.248150 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:04.248154 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:04.249873 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
W20260925 02:32:04.250088 19511 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:04.245514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:45591 (local address 127.10.13.62:46303)
0925 02:32:04.245558 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:04.245560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:04.245639 (+ 79us) server_negotiation.cc:407] Connection header received
0925 02:32:04.245697 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:04.245698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:04.245726 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:04.245784 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:04.246205 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.246715 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.247414 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.247607 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.247661 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:04.250006 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:45591: BlockingRecv error: recv got EOF from 127.10.13.2:45591 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 02:32:04 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:04 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:04.398200 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:04.392596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:50297)
0925 02:32:04.392810 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:04.392823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:04.392963 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:04.393158 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:04.393160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:04.393357 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:04.393408 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:04.393410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.394215 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.394217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:04.394819 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:04.394821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.395220 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.395221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:04.395284 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:04.395841 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:04.395846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:04.398129 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":113,"threads_started":1}
W20260925 02:32:04.398319 19511 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:04.392706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:50297 (local address 127.10.13.62:46303)
0925 02:32:04.392791 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:04.392793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:04.392915 (+ 122us) server_negotiation.cc:407] Connection header received
0925 02:32:04.393051 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:04.393052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:04.393068 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:04.393151 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:04.393504 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.394092 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.394917 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:04.395136 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:04.395220 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:04.398215 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:50297: BlockingRecv error: recv got EOF from 127.10.13.3:50297 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 02:32:05 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:05 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:05.101500 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:05.096664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:48097)
0925 02:32:05.096842 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:05.096862 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:05.096948 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:05.097174 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:05.097176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:05.097361 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:05.097414 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:05.097416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.098079 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.098080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:05.098617 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:05.098618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.099032 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.099032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:05.099082 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:05.099608 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:05.099613 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:05.101437 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
W20260925 02:32:05.101676 19515 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:05.096745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:48097 (local address 127.10.13.62:46303)
0925 02:32:05.096908 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:05.096910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:05.096940 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:32:05.097045 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:05.097046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:05.097082 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:05.097147 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:05.097489 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.097994 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.098719 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.098952 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.099020 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:05.101563 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:48097: BlockingRecv error: recv got EOF from 127.10.13.1:48097 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
Sep 25 02:32:05 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:05 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:05.255470 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:05.250430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:46443)
0925 02:32:05.250590 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:05.250603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:05.250669 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:05.250838 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:05.250838 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:05.251000 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:05.251057 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:05.251079 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.252104 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.252105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:05.252626 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:05.252627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.253019 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.253020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:05.253078 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:05.253525 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:05.253530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:05.255406 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
W20260925 02:32:05.255577 19515 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:05.250498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:46443 (local address 127.10.13.62:46303)
0925 02:32:05.250594 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:05.250597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:05.250721 (+ 124us) server_negotiation.cc:407] Connection header received
0925 02:32:05.250755 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:05.250756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:05.250770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:05.250847 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:05.251254 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.252021 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.252720 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.252933 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.253002 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:05.255504 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:46443: BlockingRecv error: recv got EOF from 127.10.13.2:46443 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 02:32:05 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:05 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:05.403568 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:05.398786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:33939)
0925 02:32:05.398955 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:05.398967 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:05.399037 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:05.399190 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:05.399191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:05.399422 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:05.399487 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:05.399489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.400170 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.400171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:05.400686 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:05.400688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.401106 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.401107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:05.401151 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:05.401566 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:05.401570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:05.403483 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
W20260925 02:32:05.403653 19515 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:05.398868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:33939 (local address 127.10.13.62:46303)
0925 02:32:05.398950 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:05.398952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:05.399040 (+ 88us) server_negotiation.cc:407] Connection header received
0925 02:32:05.399098 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:05.399099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:05.399114 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:05.399170 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:05.399585 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.400100 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.400762 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:05.401034 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:05.401096 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:05.403575 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:33939: BlockingRecv error: recv got EOF from 127.10.13.3:33939 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 02:32:06 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:06 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:06.107092 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:06.102106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:45149)
0925 02:32:06.102282 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:06.102293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:06.102343 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:06.102528 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:06.102539 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:06.102705 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:06.102759 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:06.102761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.103471 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.103472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:06.104034 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:06.104037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.104407 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.104408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:06.104455 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:06.105079 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:06.105084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:06.107013 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
W20260925 02:32:06.107247 19519 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:06.102176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:45149 (local address 127.10.13.62:46303)
0925 02:32:06.102354 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:06.102356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:06.102388 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:32:06.102415 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:06.102416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:06.102429 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:06.102512 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:06.102834 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.103392 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.104120 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.104335 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.104394 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:06.107146 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:45149: BlockingRecv error: recv got EOF from 127.10.13.1:45149 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
Sep 25 02:32:06 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:06 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:06.266117 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:06.256012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:42927)
0925 02:32:06.256182 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:06.256202 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:06.256265 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:06.256410 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:06.256411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:06.256587 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:06.256638 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:06.256639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.257324 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.257324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:06.257849 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:06.257851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.258255 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.258256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:06.258298 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:06.263987 (+ 5689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:06.263993 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:06.266044 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
W20260925 02:32:06.266296 19519 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:06.256107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:42927 (local address 127.10.13.62:46303)
0925 02:32:06.256201 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:06.256203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:06.256287 (+ 84us) server_negotiation.cc:407] Connection header received
0925 02:32:06.256317 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:06.256317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:06.256330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:06.256388 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:06.256710 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.257257 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.257930 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.258142 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.258219 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:06.266216 (+ 7997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:42927: BlockingRecv error: recv got EOF from 127.10.13.2:42927 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 02:32:06 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:06 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:06.409072 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:06.404036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:43743)
0925 02:32:06.404213 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:06.404225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:06.404293 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:06.404432 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:06.404433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:06.404606 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:06.404659 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:06.404660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.405611 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.405612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:06.406216 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:06.406219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.406542 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.406543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:06.406583 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:06.407109 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:06.407115 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:06.409001 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":56,"threads_started":1}
W20260925 02:32:06.409255 19519 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:06.404152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:43743 (local address 127.10.13.62:46303)
0925 02:32:06.404228 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:06.404230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:06.404293 (+ 63us) server_negotiation.cc:407] Connection header received
0925 02:32:06.404350 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:06.404351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:06.404365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:06.404407 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:06.404740 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.405544 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.406284 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:06.406475 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:06.406529 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:06.409180 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:43743: BlockingRecv error: recv got EOF from 127.10.13.3:43743 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 02:32:07 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:07 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:07.112207 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:07.107622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:41007)
0925 02:32:07.107803 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:07.107814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:07.107873 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:07.108054 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:07.108055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:07.108221 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:07.108275 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:07.108276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.108962 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.108963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:07.109497 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:07.109498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.109844 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.109845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:07.109887 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:07.110416 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:07.110421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:07.112147 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
W20260925 02:32:07.112346 19523 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:07.107739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:41007 (local address 127.10.13.62:46303)
0925 02:32:07.107890 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:07.107892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:07.107915 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:32:07.107940 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:07.107940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:07.107953 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:07.108003 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:07.108344 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.108850 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.109578 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.109776 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.109842 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:07.112249 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:41007: BlockingRecv error: recv got EOF from 127.10.13.1:41007 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
Sep 25 02:32:07 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:07 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:07.271661 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:07.266624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:60623)
0925 02:32:07.266811 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:07.266823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:07.266900 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:07.267110 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:07.267111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:07.267275 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:07.267353 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:07.267355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.268182 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.268183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:07.268722 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:07.268724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.269167 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.269169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:07.269234 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:07.269673 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:07.269677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:07.271559 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
W20260925 02:32:07.271720 19523 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:07.266749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:60623 (local address 127.10.13.62:46303)
0925 02:32:07.266816 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:07.266818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:07.266900 (+ 82us) server_negotiation.cc:407] Connection header received
0925 02:32:07.267017 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:07.267018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:07.267030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:07.267087 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:07.267429 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.268101 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.268813 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.269081 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.269154 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:07.271654 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:60623: BlockingRecv error: recv got EOF from 127.10.13.2:60623 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 02:32:07 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:07 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:07.414369 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:07.409682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:36605)
0925 02:32:07.409823 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:07.409849 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:07.409913 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:07.410090 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:07.410091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:07.410330 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:07.410390 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:07.410392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.411035 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.411035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:07.411558 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:07.411559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.411942 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.411943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:07.411983 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:07.412488 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:07.412493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:07.414308 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260925 02:32:07.414479 19523 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:07.409883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:36605 (local address 127.10.13.62:46303)
0925 02:32:07.409949 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:07.409952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:07.409961 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:32:07.409987 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:07.409988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:07.410003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:07.410076 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:07.410466 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.410968 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.411628 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:07.411875 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:07.411927 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:07.414402 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:36605: BlockingRecv error: recv got EOF from 127.10.13.3:36605 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 02:32:08 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:08 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:08.117825 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:08.112843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:40303)
0925 02:32:08.113037 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:08.113072 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:08.113129 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:08.113286 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:08.113287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:08.113458 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:08.113508 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:08.113510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.114180 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.114181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:08.114716 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:08.114718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.115107 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.115108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:08.115149 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:08.115683 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:08.115688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:08.117704 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":134,"threads_started":1}
W20260925 02:32:08.117875 19527 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:08.112934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:40303 (local address 127.10.13.62:46303)
0925 02:32:08.113075 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:08.113080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:08.113113 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:32:08.113188 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:08.113189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:08.113201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:08.113269 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:08.113577 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.114100 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.114806 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.115031 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.115094 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:08.117799 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:40303: BlockingRecv error: recv got EOF from 127.10.13.1:40303 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
Sep 25 02:32:08 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:08 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:08.276991 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:08.272238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:43879)
0925 02:32:08.272400 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:08.272412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:08.272503 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:08.272659 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:08.272660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:08.272848 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:08.272947 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:08.272950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.273619 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.273620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:08.274174 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:08.274175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.274554 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.274555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:08.274614 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:08.275092 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:08.275097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:08.276859 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
W20260925 02:32:08.277177 19527 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:08.272276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:43879 (local address 127.10.13.62:46303)
0925 02:32:08.272367 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:08.272369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:08.272491 (+ 122us) server_negotiation.cc:407] Connection header received
0925 02:32:08.272569 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:08.272570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:08.272585 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:08.272652 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:08.273014 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.273528 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.274262 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.274461 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.274547 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:08.277044 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:43879: BlockingRecv error: recv got EOF from 127.10.13.2:43879 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 02:32:08 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:08 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:08.419857 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:08.414862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:44453)
0925 02:32:08.415050 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:08.415062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:08.415128 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:08.415282 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:08.415283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:08.415450 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:08.415499 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:08.415501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.416331 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.416332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:08.416889 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:08.416891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.417323 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.417324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:08.417381 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:08.417810 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:08.417814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:08.419780 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
W20260925 02:32:08.419981 19527 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:08.414905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:44453 (local address 127.10.13.62:46303)
0925 02:32:08.414969 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:08.414971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:08.415134 (+ 163us) server_negotiation.cc:407] Connection header received
0925 02:32:08.415190 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:08.415190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:08.415212 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:08.415261 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:08.415632 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.416228 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.416994 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:08.417212 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:08.417303 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:08.419878 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:44453: BlockingRecv error: recv got EOF from 127.10.13.3:44453 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 02:32:09 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:09 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:09.123898 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:09.118469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:37545)
0925 02:32:09.118731 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:09.118744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:09.118809 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:09.119000 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:09.119002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:09.119183 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:09.119234 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:09.119235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.120059 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.120060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:09.120648 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:09.120651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.121050 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.121052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:09.121103 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:09.121776 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:09.121783 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:09.123821 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
W20260925 02:32:09.124102 19531 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:09.118497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:37545 (local address 127.10.13.62:46303)
0925 02:32:09.118656 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:09.118659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:09.118810 (+ 151us) server_negotiation.cc:407] Connection header received
0925 02:32:09.118855 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:09.118856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:09.118868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:09.118986 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:09.119361 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.119956 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.120736 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.120984 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.121035 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:09.123951 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:37545: BlockingRecv error: recv got EOF from 127.10.13.1:37545 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
Sep 25 02:32:09 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:09 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:09.282330 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:09.277558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:57137)
0925 02:32:09.277738 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:09.277750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:09.277822 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:09.277962 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:09.277962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:09.278129 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:09.278178 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:09.278180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.278908 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.278909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:09.279476 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:09.279478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.279854 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.279855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:09.279901 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:09.280408 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:09.280413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:09.282258 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
W20260925 02:32:09.282464 19531 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:09.277634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:57137 (local address 127.10.13.62:46303)
0925 02:32:09.277730 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:09.277732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:09.277813 (+ 81us) server_negotiation.cc:407] Connection header received
0925 02:32:09.277874 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:09.277875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:09.277887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:09.277937 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:09.278275 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.278803 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.279569 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.279768 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.279835 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:09.282378 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:57137: BlockingRecv error: recv got EOF from 127.10.13.2:57137 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 02:32:09 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:09 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:09.425216 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:09.420351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:47311)
0925 02:32:09.420493 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:09.420506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:09.420566 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:09.420754 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:09.420755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:09.421014 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:09.421068 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:09.421070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.421756 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.421757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:09.422365 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:09.422367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.422736 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.422737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:09.422787 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:09.423351 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:09.423356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:09.425142 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
W20260925 02:32:09.425403 19531 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:09.420454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:47311 (local address 127.10.13.62:46303)
0925 02:32:09.420538 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:09.420540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:09.420553 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:32:09.420618 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:09.420619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:09.420633 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:09.420701 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:09.421165 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.421663 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.422454 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:09.422657 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:09.422717 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:09.425331 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:47311: BlockingRecv error: recv got EOF from 127.10.13.3:47311 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 02:32:10 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:10 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:10.129268 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:10.124402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:54409)
0925 02:32:10.124563 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:10.124575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:10.124626 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:10.124791 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:10.124792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:10.124986 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:10.125047 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:10.125049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.125709 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.125710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:10.126257 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:10.126259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.126611 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.126612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:10.126655 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:10.127144 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:10.127150 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:10.129194 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
W20260925 02:32:10.129390 19535 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:10.124500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:54409 (local address 127.10.13.62:46303)
0925 02:32:10.124651 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:10.124653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:10.124662 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:32:10.124695 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:10.124695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:10.124707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:10.124770 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:10.125121 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.125638 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.126341 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.126540 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.126594 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:10.129311 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:54409: BlockingRecv error: recv got EOF from 127.10.13.1:54409 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
Sep 25 02:32:10 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:10 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:10.287689 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:10.282855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:46835)
0925 02:32:10.283027 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:10.283038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:10.283097 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:10.283305 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:10.283307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:10.283584 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:10.283647 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:10.283650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.284416 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.284417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:10.284980 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:10.284982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.285364 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.285365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:10.285407 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:10.285879 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:10.285883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:10.287615 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
W20260925 02:32:10.287786 19535 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:10.282986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:46835 (local address 127.10.13.62:46303)
0925 02:32:10.283065 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:10.283067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:10.283079 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:32:10.283172 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:10.283173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:10.283185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:10.283269 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:10.283737 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.284338 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.285100 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.285291 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.285348 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:10.287716 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:46835: BlockingRecv error: recv got EOF from 127.10.13.2:46835 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 02:32:10 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:10 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:10.430928 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:10.425757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:34239)
0925 02:32:10.425926 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:10.425939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:10.426001 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:10.426195 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:10.426196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:10.426374 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:10.426444 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:10.426445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.427268 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.427270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:10.427990 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:10.427993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.428347 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.428348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:10.428399 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:10.428940 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:10.428946 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:10.430860 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
W20260925 02:32:10.431092 19535 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:10.425879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:34239 (local address 127.10.13.62:46303)
0925 02:32:10.425955 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:10.425957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:10.426030 (+ 73us) server_negotiation.cc:407] Connection header received
0925 02:32:10.426065 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:10.426067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:10.426084 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:10.426159 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:10.426555 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.427187 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.428070 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:10.428256 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:10.428312 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:10.430963 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:34239: BlockingRecv error: recv got EOF from 127.10.13.3:34239 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 02:32:11 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:11 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:11.134974 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:11.129878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:37917)
0925 02:32:11.130057 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:11.130071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:11.130141 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:11.130335 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:11.130336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:11.130609 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:11.130674 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:11.130676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.131384 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.131385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:11.131955 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:11.131957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.132364 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.132365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:11.132414 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:11.132969 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:11.132975 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:11.134892 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
W20260925 02:32:11.135133 19539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:11.129941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:37917 (local address 127.10.13.62:46303)
0925 02:32:11.130057 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:11.130060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:11.130123 (+ 63us) server_negotiation.cc:407] Connection header received
0925 02:32:11.130219 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:11.130220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:11.130235 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:11.130314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:11.130748 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.131301 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.132058 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.132269 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.132339 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:11.135044 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:37917: BlockingRecv error: recv got EOF from 127.10.13.1:37917 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
Sep 25 02:32:11 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:11 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:11.298887 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:11.288245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:52767)
0925 02:32:11.288397 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:11.288407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:11.288466 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:11.288591 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:11.288592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:11.288759 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:11.288808 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:11.288810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.289605 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.289606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:11.290239 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:11.290242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.290640 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.290641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:11.290684 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:11.296704 (+ 6020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:11.296711 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:11.298822 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
W20260925 02:32:11.299081 19539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:11.288329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:52767 (local address 127.10.13.62:46303)
0925 02:32:11.288405 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:11.288407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:11.288453 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:32:11.288517 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:11.288518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:11.288529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:11.288585 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:11.288905 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.289521 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.290339 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.290566 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.290632 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:11.298969 (+ 8337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:52767: BlockingRecv error: recv got EOF from 127.10.13.2:52767 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 02:32:11 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:11 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.3@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:11.436257 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:11.431447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.3:42893)
0925 02:32:11.431637 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:11.431649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:11.431717 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:11.431865 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:11.431866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:11.432046 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:11.432095 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:11.432096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.432782 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.432782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:11.433399 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:11.433401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.433783 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.433784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:11.433836 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:11.434408 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:11.434413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:11.436173 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
W20260925 02:32:11.436419 19539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:11.431583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:42893 (local address 127.10.13.62:46303)
0925 02:32:11.431657 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:11.431659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:11.431700 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:32:11.431775 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:11.431776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:11.431789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:11.431840 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:11.432181 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.432708 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.433481 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:11.433682 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:11.433750 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:11.436334 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.3:42893: BlockingRecv error: recv got EOF from 127.10.13.3:42893 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 02:32:12 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:12 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.1@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:12.140707 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.135729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.1:40167)
0925 02:32:12.135980 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:12.135996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:12.136110 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:12.136302 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:12.136303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:12.136471 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:12.136526 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.136528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.137265 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.137266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.137795 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.137796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.138176 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.138177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.138216 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.138783 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:12.138788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:12.140636 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":125,"threads_started":1}
W20260925 02:32:12.140861 19543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:12.135915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:40167 (local address 127.10.13.62:46303)
0925 02:32:12.136152 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:12.136154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:12.136165 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:32:12.136194 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:12.136195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:12.136208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:12.136277 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:12.136602 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.137193 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.137896 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.138103 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.138165 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.140758 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.1:40167: BlockingRecv error: recv got EOF from 127.10.13.1:40167 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":115,"threads_started":1}
Sep 25 02:32:12 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:32:12 dist-test-slave-q8mg krb5kdc[18485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.13.2@KRBTEST.COM for oryx/127.10.13.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:32:12.304612 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.299495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:46303 (local address 127.10.13.2:42047)
0925 02:32:12.299669 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:12.299680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:12.299742 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:12.299938 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:12.299939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:12.300105 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:12.300157 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.300159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.300848 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.300850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.301431 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.301432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.301852 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.301853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.301893 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.302469 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:12.302475 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:12.304539 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.13.62:46303: Server oryx/127.10.13.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
W20260925 02:32:12.304715 19543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:12.299576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:42047 (local address 127.10.13.62:46303)
0925 02:32:12.299659 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:12.299663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:12.299790 (+ 127us) server_negotiation.cc:407] Connection header received
0925 02:32:12.299837 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:12.299838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:12.299852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:12.299916 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:12.300255 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.300767 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.301534 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.301765 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.301845 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.304648 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.13.2:42047: BlockingRecv error: recv got EOF from 127.10.13.2:42047 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:32:12.357314 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 19068
I20260925 02:32:12.362885 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 19220
I20260925 02:32:12.368084 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 19352
I20260925 02:32:12.372432 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 18998
2026-09-25T02:32:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17549 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T02:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:32:12Z Disabled control of system clock
I20260925 02:32:12.401818 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39835
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:32:12.485908 19548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:12.486126 19548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:12.486150 19548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:12.487488 19548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:32:12.487533 19548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:12.487547 19548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:12.487589 19548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:32:12.487627 19548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:32:12.489212 19548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:12.489490 19548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:12.489835 19548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:12.495332 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:12.495357 19548 server_base.cc:1034] running on GCE node
W20260925 02:32:12.495326 19553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:12.495324 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:12.495847 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:12.496187 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:12.497340 19548 hybrid_clock.cc:648] HybridClock initialized: now 1790303532497313 us; error 38 us; skew 500 ppm
I20260925 02:32:12.498690 19548 webserver.cc:460] Webserver started at http://127.10.13.62:40745/ using document root <none> and password file <none>
I20260925 02:32:12.498986 19548 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:12.499037 19548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:12.499229 19548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:12.500406 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "9a759f8ac4754bbcb8dd6553d3663235"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "740d3bb4a4a32d42eeddaa8ad30df5db"
server_key_iv: "435c9d10f89711f1cde91ec1cc4a917d"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.500793 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "9a759f8ac4754bbcb8dd6553d3663235"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "740d3bb4a4a32d42eeddaa8ad30df5db"
server_key_iv: "435c9d10f89711f1cde91ec1cc4a917d"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.502279 19548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:32:12.503120 19562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.503301 19548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:32:12.503422 19548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "9a759f8ac4754bbcb8dd6553d3663235"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "740d3bb4a4a32d42eeddaa8ad30df5db"
server_key_iv: "435c9d10f89711f1cde91ec1cc4a917d"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.503505 19548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:12.517210 19548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:12.517554 19548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:12.517720 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:12.522142 19548 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:39835
I20260925 02:32:12.522174 19614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:39835 every 8 connection(s)
I20260925 02:32:12.522544 19548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:32:12.523343 19615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:32:12.526116 19615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235: Bootstrap starting.
I20260925 02:32:12.526408 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 19548
I20260925 02:32:12.526523 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:32:12.526741 10292 external_mini_cluster.cc:1428] Setting key 5e27119e8e890768c4f780a0f927dff1
I20260925 02:32:12.526960 19615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235: Neither blocks nor log segments found. Creating new log.
I20260925 02:32:12.527472 19615 log.cc:824] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235: Log is configured to *not* fsync() on all Append() calls
I20260925 02:32:12.528213 19615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235: No bootstrap required, opened a new log
I20260925 02:32:12.529673 19615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a759f8ac4754bbcb8dd6553d3663235" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39835 } }
I20260925 02:32:12.529850 19615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:32:12.529901 19615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a759f8ac4754bbcb8dd6553d3663235, State: Initialized, Role: FOLLOWER
I20260925 02:32:12.530028 19615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a759f8ac4754bbcb8dd6553d3663235" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39835 } }
I20260925 02:32:12.530115 19615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:32:12.530139 19615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:32:12.530202 19615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:32:12.531024 19615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a759f8ac4754bbcb8dd6553d3663235" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39835 } }
I20260925 02:32:12.531157 19615 leader_election.cc:302] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a759f8ac4754bbcb8dd6553d3663235; no voters:
I20260925 02:32:12.531376 19615 leader_election.cc:288] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:32:12.531409 19620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:32:12.531701 19615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:32:12.532063 19620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [term 1 LEADER]: Becoming Leader. State: Replica: 9a759f8ac4754bbcb8dd6553d3663235, State: Running, Role: LEADER
I20260925 02:32:12.532212 19620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a759f8ac4754bbcb8dd6553d3663235" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39835 } }
I20260925 02:32:12.532943 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.527324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.10.13.62:39835)
0925 02:32:12.527544 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:12.527554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:12.527591 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:32:12.528071 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:12.528078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:12.528147 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:12.528308 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:12.528558 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.529534 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.530584 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.531032 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.531195 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.532040 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:12.532045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:12.532045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:32:12.532267 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:12.532309 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:32:12.532320 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:12.532405 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:12.532582 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 02:32:12.532677 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260925 02:32:12.533380 19621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a759f8ac4754bbcb8dd6553d3663235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a759f8ac4754bbcb8dd6553d3663235" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39835 } } }
I20260925 02:32:12.533378 19622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a759f8ac4754bbcb8dd6553d3663235. Latest consensus state: current_term: 1 leader_uuid: "9a759f8ac4754bbcb8dd6553d3663235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a759f8ac4754bbcb8dd6553d3663235" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39835 } } }
I20260925 02:32:12.533504 19621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [sys.catalog]: This master's current role is: LEADER
I20260925 02:32:12.533512 19622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235 [sys.catalog]: This master's current role is: LEADER
I20260925 02:32:12.533741 19630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:32:12.534262 19630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:32:12.535785 19630 catalog_manager.cc:1347] Generated new cluster ID: 305d42016f074ace9b8d6768bd375965
I20260925 02:32:12.535840 19630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:32:12.552093 19630 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:32:12.552930 19630 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:32:12.558476 19630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a759f8ac4754bbcb8dd6553d3663235: Generated new TSK 0
I20260925 02:32:12.558786 19630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:32:12.570667 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39835
--builtin_ntp_servers=127.10.13.20:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:32:12.654302 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:12.654507 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:12.654528 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:12.655740 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:12.655781 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:12.655804 19639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:32:12.657219 19639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:12.657461 19639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:12.657788 19639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:12.663545 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:12.663574 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:12.663722 19639 server_base.cc:1034] running on GCE node
W20260925 02:32:12.663745 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:12.664211 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:12.664486 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:12.665702 19639 hybrid_clock.cc:648] HybridClock initialized: now 1790303532665680 us; error 34 us; skew 500 ppm
I20260925 02:32:12.667245 19639 webserver.cc:460] Webserver started at http://127.10.13.1:38733/ using document root <none> and password file <none>
I20260925 02:32:12.667523 19639 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:12.667572 19639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:12.667712 19639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:12.668771 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "151f4f071f694ccc874ba30373ae8061"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "2259ff5caa2bf26dfe20d41ad3851b56"
server_key_iv: "c2e13b3fa1505babf5d1201a227401a8"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.669200 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "151f4f071f694ccc874ba30373ae8061"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "2259ff5caa2bf26dfe20d41ad3851b56"
server_key_iv: "c2e13b3fa1505babf5d1201a227401a8"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.670472 19639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:32:12.671183 19653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.671375 19639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:32:12.671446 19639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "151f4f071f694ccc874ba30373ae8061"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "2259ff5caa2bf26dfe20d41ad3851b56"
server_key_iv: "c2e13b3fa1505babf5d1201a227401a8"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.671538 19639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:12.685411 19639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:12.685779 19639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:12.685937 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:12.686187 19639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:12.686534 19639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:12.686573 19639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.686652 19639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:12.686681 19639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.693120 19639 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:33369
I20260925 02:32:12.693197 19766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:33369 every 8 connection(s)
I20260925 02:32:12.693547 19639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:32:12.695766 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 19639
I20260925 02:32:12.695837 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:32:12.696048 10292 external_mini_cluster.cc:1428] Setting key 0873d5768001d847d40afe30f9af317c
I20260925 02:32:12.697063 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39835
--builtin_ntp_servers=127.10.13.20:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:32:12.699535 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.694386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:47885 (local address 127.10.13.62:39835)
0925 02:32:12.694457 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:12.694459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:12.694724 (+ 265us) server_negotiation.cc:407] Connection header received
0925 02:32:12.695278 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:12.695279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:12.695288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:12.695350 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:12.696117 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.696644 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.697876 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.698114 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.698192 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.699197 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:12.699199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:12.699199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:32:12.699205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:12.699223 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:32:12.699228 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:12.699288 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:12.699382 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 02:32:12.699410 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:32:12.699649 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.694366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39835 (local address 127.10.13.1:47885)
0925 02:32:12.694590 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:12.694632 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:12.695188 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:12.695407 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:12.695410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:12.695420 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:12.696005 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.696011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.696753 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.696754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.697780 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.697782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.698228 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.698230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.698402 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.699016 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:32:12.699016 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:12.699079 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:32:12.699104 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:12.699295 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:12.699300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:12.699303 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:32:12.699358 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:32:12.699444 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":108,"threads_started":1}
I20260925 02:32:12.700184 19767 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39835
I20260925 02:32:12.700307 19767 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:12.700501 19767 heartbeater.cc:511] Master 127.10.13.62:39835 requested a full tablet report, sending...
I20260925 02:32:12.701134 19579 ts_manager.cc:194] Registered new tserver with Master: 151f4f071f694ccc874ba30373ae8061 (127.10.13.1:33369)
I20260925 02:32:12.702152 19579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.1:47885
W20260925 02:32:12.781009 19770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:12.781214 19770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:12.781235 19770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:12.782369 19770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:12.782403 19770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:12.782426 19770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:32:12.783682 19770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:12.783870 19770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:12.784124 19770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:12.789301 19776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:12.789283 19775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:12.789271 19778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:12.789619 19770 server_base.cc:1034] running on GCE node
I20260925 02:32:12.789860 19770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:12.790119 19770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:12.791275 19770 hybrid_clock.cc:648] HybridClock initialized: now 1790303532791252 us; error 32 us; skew 500 ppm
I20260925 02:32:12.792752 19770 webserver.cc:460] Webserver started at http://127.10.13.2:46235/ using document root <none> and password file <none>
I20260925 02:32:12.793062 19770 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:12.793114 19770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:12.793272 19770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:12.794488 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "9575fa8b4f5e4b4ea0d038778dc22f00"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "acb7e85aea0fe979c85505671f9aec3b"
server_key_iv: "76c6aa4ae80b005d0e23cf9799de6d2a"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.794865 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "9575fa8b4f5e4b4ea0d038778dc22f00"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "acb7e85aea0fe979c85505671f9aec3b"
server_key_iv: "76c6aa4ae80b005d0e23cf9799de6d2a"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.796216 19770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:32:12.797003 19784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.797259 19770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:32:12.797372 19770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "9575fa8b4f5e4b4ea0d038778dc22f00"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "acb7e85aea0fe979c85505671f9aec3b"
server_key_iv: "76c6aa4ae80b005d0e23cf9799de6d2a"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.797451 19770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:12.806384 19770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:12.806736 19770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:12.806915 19770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:12.807125 19770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:12.807476 19770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:12.807533 19770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.807583 19770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:12.807611 19770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.813508 19770 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:46349
I20260925 02:32:12.813551 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:46349 every 8 connection(s)
I20260925 02:32:12.813884 19770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:32:12.819235 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.814656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:40057 (local address 127.10.13.62:39835)
0925 02:32:12.814718 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:12.814720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:12.814974 (+ 254us) server_negotiation.cc:407] Connection header received
0925 02:32:12.815557 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:12.815558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:12.815572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:12.815622 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:12.816305 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.816826 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.817847 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.818057 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.818135 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.818920 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:12.818922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:12.818922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:32:12.818927 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:12.818941 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:32:12.818945 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:12.818999 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:12.819124 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 02:32:12.819152 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:32:12.819460 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.814646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39835 (local address 127.10.13.2:40057)
0925 02:32:12.814859 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:12.814892 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:12.815432 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:12.815698 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:12.815702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:12.815711 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:12.816212 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.816216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.816965 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.816966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.817755 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.817758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.818145 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.818146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.818279 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.818769 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:32:12.818769 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:12.818799 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:32:12.818821 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:12.819005 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:12.819011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:12.819014 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:32:12.819125 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:32:12.819228 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260925 02:32:12.819964 19898 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39835
I20260925 02:32:12.820070 19898 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:12.820258 19898 heartbeater.cc:511] Master 127.10.13.62:39835 requested a full tablet report, sending...
I20260925 02:32:12.820739 19579 ts_manager.cc:194] Registered new tserver with Master: 9575fa8b4f5e4b4ea0d038778dc22f00 (127.10.13.2:46349)
I20260925 02:32:12.821184 19579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.2:40057
I20260925 02:32:12.822633 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 19770
I20260925 02:32:12.822714 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:32:12.822872 10292 external_mini_cluster.cc:1428] Setting key 869dc270c025c353e27f2f4d35b0c611
I20260925 02:32:12.823832 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39835
--builtin_ntp_servers=127.10.13.20:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:32:12.905921 19901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:12.906186 19901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:12.906224 19901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:12.907461 19901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:12.907529 19901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:12.907569 19901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:32:12.908981 19901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:12.909253 19901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:12.909565 19901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:12.914824 19909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:12.914834 19906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:12.914839 19907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:12.915112 19901 server_base.cc:1034] running on GCE node
I20260925 02:32:12.915459 19901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:12.915735 19901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:12.916929 19901 hybrid_clock.cc:648] HybridClock initialized: now 1790303532916904 us; error 56 us; skew 500 ppm
I20260925 02:32:12.918342 19901 webserver.cc:460] Webserver started at http://127.10.13.3:41387/ using document root <none> and password file <none>
I20260925 02:32:12.918618 19901 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:12.918694 19901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:12.918882 19901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:12.919971 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "26a9b72694f5470a9d4d9ccb8eb66f7a"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "5b33fc87b7cd25b21e765f60e33c08e0"
server_key_iv: "fd2e01e6b4412721b5b4c18a3d43df87"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.920332 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "26a9b72694f5470a9d4d9ccb8eb66f7a"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "5b33fc87b7cd25b21e765f60e33c08e0"
server_key_iv: "fd2e01e6b4412721b5b4c18a3d43df87"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.921787 19901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:32:12.922582 19915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.922767 19901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:32:12.922879 19901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "26a9b72694f5470a9d4d9ccb8eb66f7a"
format_stamp: "Formatted at 2026-09-25 02:32:12 on dist-test-slave-q8mg"
server_key: "5b33fc87b7cd25b21e765f60e33c08e0"
server_key_iv: "fd2e01e6b4412721b5b4c18a3d43df87"
server_key_version: "encryptionkey@0"
I20260925 02:32:12.922956 19901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:12.933705 19901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:12.934051 19901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:12.934211 19901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:12.934446 19901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:12.934787 19901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:12.934842 19901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.934876 19901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:12.934902 19901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:12.940949 20028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:39731 every 8 connection(s)
I20260925 02:32:12.940949 19901 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:39731
I20260925 02:32:12.941388 19901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:32:12.946908 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.942225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:55307 (local address 127.10.13.62:39835)
0925 02:32:12.942278 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:12.942280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:12.942492 (+ 212us) server_negotiation.cc:407] Connection header received
0925 02:32:12.943069 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:12.943070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:12.943112 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:12.943193 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:12.943930 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.944434 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.945463 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.945659 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.945746 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.946592 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:12.946594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:12.946595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:32:12.946600 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:12.946618 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:32:12.946622 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:12.946694 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:12.946793 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:32:12.946823 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:32:12.947124 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:12.942172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39835 (local address 127.10.13.3:55307)
0925 02:32:12.942361 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:12.942396 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:12.942948 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:12.943300 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:12.943304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:12.943313 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:12.943825 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.943829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.944534 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.944535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.945363 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:12.945365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:12.945756 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:12.945757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:12.945898 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:12.946446 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:32:12.946446 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:12.946476 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:32:12.946498 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:12.946695 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:12.946702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:12.946705 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:32:12.946794 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:32:12.946902 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260925 02:32:12.947649 20029 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39835
I20260925 02:32:12.947755 20029 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:12.947943 20029 heartbeater.cc:511] Master 127.10.13.62:39835 requested a full tablet report, sending...
I20260925 02:32:12.948336 19579 ts_manager.cc:194] Registered new tserver with Master: 26a9b72694f5470a9d4d9ccb8eb66f7a (127.10.13.3:39731)
I20260925 02:32:12.948757 19579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.3:55307
I20260925 02:32:12.948876 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 19901
I20260925 02:32:12.949012 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:32:12.949232 10292 external_mini_cluster.cc:1428] Setting key 7119d6ad9de70f98345c754ac91622ca
I20260925 02:32:12.950187 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:32:12.952064 19618 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:12.951547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.10.13.62:39835)
0925 02:32:12.951614 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:12.951617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:12.951750 (+ 133us) server_negotiation.cc:407] Connection header received
0925 02:32:12.951798 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:12.951799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:12.951809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:12.951847 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:12.951985 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46138: BlockingRecv error: recv got EOF from 127.0.0.1:46138 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:32:12.952113 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 19639
I20260925 02:32:12.958235 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 19770
I20260925 02:32:12.963373 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 19901
I20260925 02:32:12.968351 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 19548
2026-09-25T02:32:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T02:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:32:12Z Disabled control of system clock
I20260925 02:32:12.996055 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34397
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:34397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:32:13.084014 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:13.084209 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:13.084229 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:13.085292 20040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:32:13.085323 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:13.085336 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:13.085348 20040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:32:13.085361 20040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:32:13.086624 20040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:34397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:13.086827 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:13.087060 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:13.091956 20048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:13.091958 20046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:13.092130 20040 server_base.cc:1034] running on GCE node
W20260925 02:32:13.091945 20045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:13.092439 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:13.092697 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:13.093854 20040 hybrid_clock.cc:648] HybridClock initialized: now 1790303533093843 us; error 32 us; skew 500 ppm
I20260925 02:32:13.095178 20040 webserver.cc:460] Webserver started at http://127.10.13.62:42867/ using document root <none> and password file <none>
I20260925 02:32:13.095432 20040 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:13.095480 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:13.095633 20040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:13.096719 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "1d9cca816b6e4a849924660ad6a7ddfb"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "ccb93fc8446c86e0e916aed8dc17c9c4"
server_key_iv: "ab33ba18176a93737bbebb071a02af22"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.097121 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "1d9cca816b6e4a849924660ad6a7ddfb"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "ccb93fc8446c86e0e916aed8dc17c9c4"
server_key_iv: "ab33ba18176a93737bbebb071a02af22"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.098431 20040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:32:13.099246 20054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.099457 20040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:32:13.099552 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "1d9cca816b6e4a849924660ad6a7ddfb"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "ccb93fc8446c86e0e916aed8dc17c9c4"
server_key_iv: "ab33ba18176a93737bbebb071a02af22"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.099627 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:13.103060 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:13.103332 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:13.103483 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:13.107365 20040 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:34397
I20260925 02:32:13.107419 20106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:34397 every 8 connection(s)
I20260925 02:32:13.107739 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:32:13.108410 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:32:13.110710 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 20040
I20260925 02:32:13.110811 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:32:13.110985 10292 external_mini_cluster.cc:1428] Setting key e69315e26e46accac33c84f2f63de3ee
I20260925 02:32:13.111615 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb: Bootstrap starting.
I20260925 02:32:13.112334 20107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb: Neither blocks nor log segments found. Creating new log.
I20260925 02:32:13.112669 20107 log.cc:824] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:32:13.113649 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb: No bootstrap required, opened a new log
I20260925 02:32:13.113829 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:13.111503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.10.13.62:34397)
0925 02:32:13.111712 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:13.111722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:13.111754 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:32:13.112336 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:13.112345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:13.112402 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:13.112576 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:13.113295 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:13.113299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:13.113299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:32:13.113303 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:13.113341 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:32:13.113351 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:13.113436 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:13.113500 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:32:13.113583 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260925 02:32:13.115492 20107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cca816b6e4a849924660ad6a7ddfb" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34397 } }
I20260925 02:32:13.115656 20107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:32:13.115705 20107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9cca816b6e4a849924660ad6a7ddfb, State: Initialized, Role: FOLLOWER
I20260925 02:32:13.115826 20107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cca816b6e4a849924660ad6a7ddfb" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34397 } }
I20260925 02:32:13.115906 20107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:32:13.115948 20107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:32:13.115996 20107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:32:13.116506 20107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cca816b6e4a849924660ad6a7ddfb" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34397 } }
I20260925 02:32:13.116629 20107 leader_election.cc:302] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d9cca816b6e4a849924660ad6a7ddfb; no voters:
I20260925 02:32:13.116811 20107 leader_election.cc:288] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:32:13.116853 20112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:32:13.117074 20112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [term 1 LEADER]: Becoming Leader. State: Replica: 1d9cca816b6e4a849924660ad6a7ddfb, State: Running, Role: LEADER
I20260925 02:32:13.117102 20107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:32:13.117175 20112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cca816b6e4a849924660ad6a7ddfb" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34397 } }
I20260925 02:32:13.117574 20114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d9cca816b6e4a849924660ad6a7ddfb. Latest consensus state: current_term: 1 leader_uuid: "1d9cca816b6e4a849924660ad6a7ddfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cca816b6e4a849924660ad6a7ddfb" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34397 } } }
I20260925 02:32:13.117686 20114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [sys.catalog]: This master's current role is: LEADER
I20260925 02:32:13.117856 20113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d9cca816b6e4a849924660ad6a7ddfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cca816b6e4a849924660ad6a7ddfb" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34397 } } }
I20260925 02:32:13.117928 20113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb [sys.catalog]: This master's current role is: LEADER
I20260925 02:32:13.118306 20121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:32:13.118836 20121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:32:13.120432 20121 catalog_manager.cc:1347] Generated new cluster ID: ef34e0b7b4cc4e739c16316741273ff6
I20260925 02:32:13.120501 20121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:32:13.147086 20121 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:32:13.147292 20121 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:32:13.154690 20121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d9cca816b6e4a849924660ad6a7ddfb: Generated new TSK 0
I20260925 02:32:13.154917 20121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:32:13.158397 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34397
--builtin_ntp_servers=127.10.13.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:32:13.259372 20131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:13.259608 20131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:13.259632 20131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:13.260892 20131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:13.260931 20131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:13.260962 20131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:32:13.262369 20131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:13.262574 20131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:13.262841 20131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:13.268605 20136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:13.268621 20137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:13.268777 20131 server_base.cc:1034] running on GCE node
W20260925 02:32:13.268868 20139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:13.269138 20131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:13.269366 20131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:13.270524 20131 hybrid_clock.cc:648] HybridClock initialized: now 1790303533270507 us; error 30 us; skew 500 ppm
I20260925 02:32:13.271862 20131 webserver.cc:460] Webserver started at http://127.10.13.1:43495/ using document root <none> and password file <none>
I20260925 02:32:13.272130 20131 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:13.272183 20131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:13.272374 20131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:13.273693 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "696c2decfb824113962550ed6a550068"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "a2e7c5add46966828d3f0ef65cb6b8cf"
server_key_iv: "295e537e55a9304b595bb8deec2a9046"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.274132 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "696c2decfb824113962550ed6a550068"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "a2e7c5add46966828d3f0ef65cb6b8cf"
server_key_iv: "295e537e55a9304b595bb8deec2a9046"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.275553 20131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:32:13.276413 20145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.276602 20131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:32:13.276722 20131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "696c2decfb824113962550ed6a550068"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "a2e7c5add46966828d3f0ef65cb6b8cf"
server_key_iv: "295e537e55a9304b595bb8deec2a9046"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.276805 20131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:13.280679 20131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:13.281032 20131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:13.281196 20131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:13.281425 20131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:13.281765 20131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:13.281826 20131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.281867 20131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:13.281898 20131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.288501 20131 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:38931
I20260925 02:32:13.288558 20258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:38931 every 8 connection(s)
I20260925 02:32:13.289069 20131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:32:13.292412 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:13.289965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:42315 (local address 127.10.13.62:34397)
0925 02:32:13.290025 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:13.290027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:13.290329 (+ 302us) server_negotiation.cc:407] Connection header received
0925 02:32:13.290852 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:13.290853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:13.290861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:13.290901 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:13.292152 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:13.292154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:13.292154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:32:13.292159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:13.292172 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:32:13.292177 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:13.292232 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:13.292327 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 02:32:13.292342 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:32:13.292642 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:13.289952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34397 (local address 127.10.13.1:42315)
0925 02:32:13.290197 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:13.290231 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:13.290734 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:13.290987 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:13.290990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:13.290998 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:13.291965 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:32:13.291965 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:13.291994 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:32:13.292023 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:13.292237 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:13.292243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:13.292246 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:32:13.292337 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:32:13.292400 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260925 02:32:13.293226 20259 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34397
I20260925 02:32:13.293305 20259 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:13.293458 20259 heartbeater.cc:511] Master 127.10.13.62:34397 requested a full tablet report, sending...
I20260925 02:32:13.294021 20071 ts_manager.cc:194] Registered new tserver with Master: 696c2decfb824113962550ed6a550068 (127.10.13.1:38931)
I20260925 02:32:13.294384 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 20131
I20260925 02:32:13.294489 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:32:13.294701 10292 external_mini_cluster.cc:1428] Setting key 88cdef87fe434ca8a71524dc769c92e5
I20260925 02:32:13.295634 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34397
--builtin_ntp_servers=127.10.13.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:32:13.378569 20262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:13.378782 20262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:13.378804 20262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:13.380051 20262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:13.380091 20262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:13.380119 20262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:32:13.381633 20262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:13.381881 20262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:13.382196 20262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:13.387253 20270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:13.387253 20267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:13.387418 20262 server_base.cc:1034] running on GCE node
W20260925 02:32:13.387321 20268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:13.387797 20262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:13.388053 20262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:13.389259 20262 hybrid_clock.cc:648] HybridClock initialized: now 1790303533389229 us; error 41 us; skew 500 ppm
I20260925 02:32:13.390672 20262 webserver.cc:460] Webserver started at http://127.10.13.2:33937/ using document root <none> and password file <none>
I20260925 02:32:13.390995 20262 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:13.391074 20262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:13.391224 20262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:13.392302 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "505998ec5ec441ccbc11f392de8605a3"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "c2b55d521b6707142cfa64b29f19ae59"
server_key_iv: "a4f1e9af20b6ea0d8b6b62b353d0c7b2"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.392673 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "505998ec5ec441ccbc11f392de8605a3"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "c2b55d521b6707142cfa64b29f19ae59"
server_key_iv: "a4f1e9af20b6ea0d8b6b62b353d0c7b2"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.394106 20262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:32:13.394914 20276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.395113 20262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:32:13.395217 20262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "505998ec5ec441ccbc11f392de8605a3"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "c2b55d521b6707142cfa64b29f19ae59"
server_key_iv: "a4f1e9af20b6ea0d8b6b62b353d0c7b2"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.395298 20262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:13.398806 20262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:13.399077 20262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:13.399216 20262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:13.399421 20262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:13.399729 20262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:13.399781 20262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.399822 20262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:13.399849 20262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.405928 20262 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:42117
I20260925 02:32:13.406011 20389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:42117 every 8 connection(s)
I20260925 02:32:13.406351 20262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:32:13.409387 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:13.407123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:52369 (local address 127.10.13.62:34397)
0925 02:32:13.407186 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:13.407188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:13.407411 (+ 223us) server_negotiation.cc:407] Connection header received
0925 02:32:13.407890 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:13.407891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:13.407896 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:13.407962 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:13.409180 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:13.409182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:13.409182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:32:13.409187 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:13.409201 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:32:13.409205 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:13.409254 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:13.409320 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 02:32:13.409332 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:32:13.409577 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:13.407086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34397 (local address 127.10.13.2:52369)
0925 02:32:13.407280 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:13.407322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:13.407796 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:13.408020 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:13.408023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:13.408032 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:13.409026 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:32:13.409026 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:13.409059 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:32:13.409089 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:13.409259 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:13.409261 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:13.409263 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:32:13.409314 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:32:13.409364 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260925 02:32:13.410123 20390 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34397
I20260925 02:32:13.410229 20390 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:13.410400 20390 heartbeater.cc:511] Master 127.10.13.62:34397 requested a full tablet report, sending...
I20260925 02:32:13.410416 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 20262
I20260925 02:32:13.410507 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:32:13.410744 10292 external_mini_cluster.cc:1428] Setting key e89f7778314d2d3e06d04e98b5338473
I20260925 02:32:13.410780 20071 ts_manager.cc:194] Registered new tserver with Master: 505998ec5ec441ccbc11f392de8605a3 (127.10.13.2:42117)
I20260925 02:32:13.411741 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34397
--builtin_ntp_servers=127.10.13.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:32:13.493799 20393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:13.494021 20393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:13.494043 20393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:13.495129 20393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:13.495163 20393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:13.495188 20393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:32:13.496460 20393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:13.496670 20393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:13.496980 20393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:13.502458 20401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:13.502478 20398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:13.502467 20399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:13.502812 20393 server_base.cc:1034] running on GCE node
I20260925 02:32:13.503077 20393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:13.503331 20393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:13.504503 20393 hybrid_clock.cc:648] HybridClock initialized: now 1790303533504490 us; error 28 us; skew 500 ppm
I20260925 02:32:13.505910 20393 webserver.cc:460] Webserver started at http://127.10.13.3:33277/ using document root <none> and password file <none>
I20260925 02:32:13.506181 20393 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:13.506230 20393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:13.506400 20393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:13.507530 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "c1526f106676405abba9d254bfc2c118"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "41d0ff1fc196e7acafeacd963c96165c"
server_key_iv: "682cbe65e0744eedf983e8bbc0461807"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.507893 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "c1526f106676405abba9d254bfc2c118"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "41d0ff1fc196e7acafeacd963c96165c"
server_key_iv: "682cbe65e0744eedf983e8bbc0461807"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.509292 20393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 02:32:13.510118 20407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.510301 20393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:32:13.510396 20393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "c1526f106676405abba9d254bfc2c118"
format_stamp: "Formatted at 2026-09-25 02:32:13 on dist-test-slave-q8mg"
server_key: "41d0ff1fc196e7acafeacd963c96165c"
server_key_iv: "682cbe65e0744eedf983e8bbc0461807"
server_key_version: "encryptionkey@0"
I20260925 02:32:13.510502 20393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:13.513962 20393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:13.514223 20393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:13.514357 20393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:13.514571 20393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:13.514870 20393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:13.514904 20393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.514948 20393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:13.514974 20393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:13.521380 20393 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:37427
I20260925 02:32:13.521421 20520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:37427 every 8 connection(s)
I20260925 02:32:13.521798 20393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:32:13.524911 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:13.522536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:52301 (local address 127.10.13.62:34397)
0925 02:32:13.522578 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:13.522580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:13.522839 (+ 259us) server_negotiation.cc:407] Connection header received
0925 02:32:13.523374 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:13.523376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:13.523383 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:13.523449 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:13.524635 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:13.524637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:13.524637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:32:13.524643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:13.524657 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:32:13.524661 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:13.524712 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:13.524804 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 02:32:13.524818 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:32:13.525123 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:13.522522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34397 (local address 127.10.13.3:52301)
0925 02:32:13.522708 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:13.522743 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:13.523248 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:13.523515 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:13.523519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:13.523529 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:13.524477 (+ 948us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:32:13.524477 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:13.524507 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:32:13.524535 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:13.524710 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:13.524715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:13.524718 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:32:13.524814 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:32:13.524896 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260925 02:32:13.525609 20521 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34397
I20260925 02:32:13.525700 20521 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:13.525820 20521 heartbeater.cc:511] Master 127.10.13.62:34397 requested a full tablet report, sending...
I20260925 02:32:13.526202 20071 ts_manager.cc:194] Registered new tserver with Master: c1526f106676405abba9d254bfc2c118 (127.10.13.3:37427)
I20260925 02:32:13.526494 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 20393
I20260925 02:32:13.526703 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:32:13.526911 10292 external_mini_cluster.cc:1428] Setting key 6bfad535ebbccd8685c0e7bc16bc3c76
I20260925 02:32:13.527607 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:32:13.529385 20110 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:32:13.528809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.10.13.62:34397)
0925 02:32:13.528945 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:13.528948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:13.529078 (+ 130us) server_negotiation.cc:407] Connection header received
0925 02:32:13.529119 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:13.529119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:13.529124 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:13.529170 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:13.529331 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42006: BlockingRecv error: recv got EOF from 127.0.0.1:42006 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:32:13.529466 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 20131
I20260925 02:32:13.535144 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 20262
I20260925 02:32:13.540333 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 20393
I20260925 02:32:13.546922 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 20040
2026-09-25T02:32:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (577 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:32:13 dist-test-slave-q8mg krb5kdc[20531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:32:13 dist-test-slave-q8mg krb5kdc[20531](info): set up 2 sockets
Sep 25 02:32:13 dist-test-slave-q8mg krb5kdc[20531](info): commencing operation
krb5kdc: starting...
W20260925 02:32:14.580363 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:32:14 dist-test-slave-q8mg krb5kdc[20531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:32:14.700209 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:44091
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:44091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:32:14.787721 20544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:14.787948 20544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:14.787971 20544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:14.789188 20544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:32:14.789229 20544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:14.789244 20544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:14.789283 20544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:32:14.789312 20544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:32:14.790789 20544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:44091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:14.791059 20544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:14.791286 20544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:14.796306 20552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:14.796301 20549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:14.796303 20550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:14.796329 20544 server_base.cc:1034] running on GCE node
I20260925 02:32:14.796784 20544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:14.797124 20544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:14.798285 20544 hybrid_clock.cc:648] HybridClock initialized: now 1790303534798263 us; error 48 us; skew 500 ppm
Sep 25 02:32:14 dist-test-slave-q8mg krb5kdc[20531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303534, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:32:14.800827 20544 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:32:14.801383 20544 webserver.cc:460] Webserver started at http://127.10.13.62:41131/ using document root <none> and password file <none>
I20260925 02:32:14.801645 20544 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:14.801690 20544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:14.801834 20544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:14.802884 20544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "3c9e932ebeb14d289bd87e83d4e0208f"
format_stamp: "Formatted at 2026-09-25 02:32:14 on dist-test-slave-q8mg"
server_key: "18e7c333f1f5ed62ec619066a60cfd46"
server_key_iv: "b37f8d61aa7c140889b47608125d8358"
server_key_version: "encryptionkey@0"
I20260925 02:32:14.803257 20544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "3c9e932ebeb14d289bd87e83d4e0208f"
format_stamp: "Formatted at 2026-09-25 02:32:14 on dist-test-slave-q8mg"
server_key: "18e7c333f1f5ed62ec619066a60cfd46"
server_key_iv: "b37f8d61aa7c140889b47608125d8358"
server_key_version: "encryptionkey@0"
I20260925 02:32:14.804567 20544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 02:32:14.805449 20559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:14.805766 20544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:32:14.805867 20544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "3c9e932ebeb14d289bd87e83d4e0208f"
format_stamp: "Formatted at 2026-09-25 02:32:14 on dist-test-slave-q8mg"
server_key: "18e7c333f1f5ed62ec619066a60cfd46"
server_key_iv: "b37f8d61aa7c140889b47608125d8358"
server_key_version: "encryptionkey@0"
I20260925 02:32:14.805944 20544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:14.812647 20544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:14.814958 20544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:14.815119 20544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:14.819167 20544 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:44091
I20260925 02:32:14.819217 20611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:44091 every 8 connection(s)
I20260925 02:32:14.819571 20544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:32:14.820573 20612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:32:14.822886 20612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f: Bootstrap starting.
I20260925 02:32:14.823495 20612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f: Neither blocks nor log segments found. Creating new log.
I20260925 02:32:14.823789 20612 log.cc:824] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:32:14.824520 20612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f: No bootstrap required, opened a new log
I20260925 02:32:14.826215 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 20544
I20260925 02:32:14.826303 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:32:14.826207 20612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9e932ebeb14d289bd87e83d4e0208f" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44091 } }
I20260925 02:32:14.826409 20612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:32:14.826479 20612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9e932ebeb14d289bd87e83d4e0208f, State: Initialized, Role: FOLLOWER
I20260925 02:32:14.826579 10292 external_mini_cluster.cc:1428] Setting key 32cde919dbdfc748c64bba4c8c26d76c
I20260925 02:32:14.826653 20612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9e932ebeb14d289bd87e83d4e0208f" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44091 } }
I20260925 02:32:14.826741 20612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:32:14.826797 20612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:32:14.826838 20612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:32:14.828012 20612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9e932ebeb14d289bd87e83d4e0208f" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44091 } }
I20260925 02:32:14.828150 20612 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9e932ebeb14d289bd87e83d4e0208f; no voters:
I20260925 02:32:14.828325 20612 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:32:14.828384 20617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:32:14.828593 20617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [term 1 LEADER]: Becoming Leader. State: Replica: 3c9e932ebeb14d289bd87e83d4e0208f, State: Running, Role: LEADER
I20260925 02:32:14.828629 20612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:32:14.828749 20617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9e932ebeb14d289bd87e83d4e0208f" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44091 } }
I20260925 02:32:14.829239 20617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9e932ebeb14d289bd87e83d4e0208f. Latest consensus state: current_term: 1 leader_uuid: "3c9e932ebeb14d289bd87e83d4e0208f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9e932ebeb14d289bd87e83d4e0208f" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44091 } } }
I20260925 02:32:14.829377 20617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [sys.catalog]: This master's current role is: LEADER
I20260925 02:32:14.829385 20619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9e932ebeb14d289bd87e83d4e0208f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9e932ebeb14d289bd87e83d4e0208f" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 44091 } } }
I20260925 02:32:14.829479 20619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f [sys.catalog]: This master's current role is: LEADER
I20260925 02:32:14.829636 20624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:32:14.830405 20624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:32:14.832319 20624 catalog_manager.cc:1347] Generated new cluster ID: 9596105efec4407ca27d915d38eda659
I20260925 02:32:14.832379 20624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:32:14 dist-test-slave-q8mg krb5kdc[20531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:32:14.839155 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:14.827179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.10.13.62:44091)
0925 02:32:14.827351 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:14.827358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:14.827390 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:32:14.827495 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:14.827500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:14.827542 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:14.827651 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:14.828499 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:14.830155 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:14.831315 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:14.831914 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:14.832024 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:14.835041 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:14.835046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:14.835046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:32:14.835059 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:14.837330 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:14.837767 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:14.837771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:14.837771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:14.837917 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:14.838158 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:14.838158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:14.838159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:14.838479 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:14.838655 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:14.838769 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 02:32:14.838923 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260925 02:32:14.843798 20624 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:32:14.844336 20624 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:32:14.851343 20624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f: Generated new TSK 0
I20260925 02:32:14.851572 20624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:32:14.888228 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44091
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:32:14.986779 20638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:14.987001 20638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:14.987028 20638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:14.988250 20638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:14.988289 20638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:14.988315 20638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:32:14.989790 20638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:14.989980 20638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:14.990195 20638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:14.995114 20646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:14.995127 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:14.995383 20638 server_base.cc:1034] running on GCE node
W20260925 02:32:14.995153 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:14.995669 20638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:14.995923 20638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:14.997088 20638 hybrid_clock.cc:648] HybridClock initialized: now 1790303534997072 us; error 26 us; skew 500 ppm
Sep 25 02:32:14 dist-test-slave-q8mg krb5kdc[20531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303534, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:32:14.999583 20638 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:32:15.000005 20638 webserver.cc:460] Webserver started at http://127.10.13.1:42517/ using document root <none> and password file <none>
I20260925 02:32:15.000257 20638 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:15.000326 20638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:15.000474 20638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:15.001597 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab"
format_stamp: "Formatted at 2026-09-25 02:32:14 on dist-test-slave-q8mg"
server_key: "8db236b77b61799af8c5b0f6ed52c7ec"
server_key_iv: "785b3b4544842e336b0b6ec59f1840c2"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.002000 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab"
format_stamp: "Formatted at 2026-09-25 02:32:14 on dist-test-slave-q8mg"
server_key: "8db236b77b61799af8c5b0f6ed52c7ec"
server_key_iv: "785b3b4544842e336b0b6ec59f1840c2"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.003504 20638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:32:15.005041 20653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.005193 20638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:32:15.005277 20638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab"
format_stamp: "Formatted at 2026-09-25 02:32:14 on dist-test-slave-q8mg"
server_key: "8db236b77b61799af8c5b0f6ed52c7ec"
server_key_iv: "785b3b4544842e336b0b6ec59f1840c2"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.005357 20638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:15.034438 20638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:15.037177 20638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:15.037349 20638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:15.037578 20638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:15.037919 20638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:15.037976 20638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.038018 20638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:15.038045 20638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.043777 20638 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:42583
I20260925 02:32:15.043862 20766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:42583 every 8 connection(s)
I20260925 02:32:15.044204 20638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:32:15 dist-test-slave-q8mg krb5kdc[20531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303534, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:32:15.053732 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.044993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:52795 (local address 127.10.13.62:44091)
0925 02:32:15.045061 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.045063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.045297 (+ 234us) server_negotiation.cc:407] Connection header received
0925 02:32:15.045794 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.045795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.045806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.045856 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:15.046796 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.047306 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.048318 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.048506 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.048560 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.050779 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:15.050782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:15.050782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:32:15.050790 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:15.052224 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:15.052655 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:15.052657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:15.052658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:15.052720 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:15.053038 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:15.053039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:15.053040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:15.053161 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:15.053240 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.053522 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.053626 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:32:15.053915 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.044997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44091 (local address 127.10.13.1:52795)
0925 02:32:15.045161 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.045201 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:15.045706 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:15.045920 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:15.045923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:15.046516 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:15.046717 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.046722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.047395 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.047395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.048240 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.048243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.048581 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.048582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.048686 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.049226 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:15.049236 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:15.050667 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:15.052326 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:32:15.052334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:32:15.052336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:32:15.052574 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:32:15.052790 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:32:15.052791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:32:15.052792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:32:15.052956 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:32:15.053232 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:15.053239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:15.053385 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 02:32:15.053487 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:32:15.053657 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260925 02:32:15.054431 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 20638
I20260925 02:32:15.054543 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:32:15.054535 20767 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44091
I20260925 02:32:15.054638 20767 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:15.054782 10292 external_mini_cluster.cc:1428] Setting key a7981c9d514b53b0d2ef9adcc778edc6
I20260925 02:32:15.054879 20767 heartbeater.cc:511] Master 127.10.13.62:44091 requested a full tablet report, sending...
I20260925 02:32:15.055464 20575 ts_manager.cc:194] Registered new tserver with Master: 0f6f3901ef5f42d0be7afd3e0fed40ab (127.10.13.1:42583)
I20260925 02:32:15.056082 20575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:52795
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:32:15.080144 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44091
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:32:15.162076 20772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:15.162315 20772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:15.162352 20772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:15.163481 20772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:15.163549 20772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:15.163595 20772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:32:15.165149 20772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:15.165419 20772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:15.165660 20772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:15.170846 20778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:15.170894 20772 server_base.cc:1034] running on GCE node
W20260925 02:32:15.170830 20777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:15.170830 20780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:15.171258 20772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:15.171505 20772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:15.172677 20772 hybrid_clock.cc:648] HybridClock initialized: now 1790303535172662 us; error 27 us; skew 500 ppm
Sep 25 02:32:15 dist-test-slave-q8mg krb5kdc[20531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303535, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:32:15.175350 20772 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:32:15.175756 20772 webserver.cc:460] Webserver started at http://127.10.13.2:38919/ using document root <none> and password file <none>
I20260925 02:32:15.175971 20772 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:15.176013 20772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:15.176107 20772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:15.177201 20772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "f797fc06b94b4aa8a543f279db0c778a"
format_stamp: "Formatted at 2026-09-25 02:32:15 on dist-test-slave-q8mg"
server_key: "440d987530130b7fc2c2dd7f76e3abca"
server_key_iv: "660f1d5bfb49b24bac0c151b61d7f803"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.177502 20772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "f797fc06b94b4aa8a543f279db0c778a"
format_stamp: "Formatted at 2026-09-25 02:32:15 on dist-test-slave-q8mg"
server_key: "440d987530130b7fc2c2dd7f76e3abca"
server_key_iv: "660f1d5bfb49b24bac0c151b61d7f803"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.178776 20772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:32:15.179472 20787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.179654 20772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.179764 20772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "f797fc06b94b4aa8a543f279db0c778a"
format_stamp: "Formatted at 2026-09-25 02:32:15 on dist-test-slave-q8mg"
server_key: "440d987530130b7fc2c2dd7f76e3abca"
server_key_iv: "660f1d5bfb49b24bac0c151b61d7f803"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.179838 20772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:15.193604 20772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:15.195854 20772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:15.196010 20772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:15.196233 20772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:15.196542 20772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:15.196596 20772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.196636 20772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:15.196662 20772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.202330 20772 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:36687
I20260925 02:32:15.202430 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:36687 every 8 connection(s)
I20260925 02:32:15.202728 20772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:32:15.205096 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 20772
I20260925 02:32:15.205180 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:32:15.205404 10292 external_mini_cluster.cc:1428] Setting key 6e27b25f1a392155e8e8f7555cc981e0
Sep 25 02:32:15 dist-test-slave-q8mg krb5kdc[20531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303535, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:32:15.213227 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.203519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:50691 (local address 127.10.13.62:44091)
0925 02:32:15.203567 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.203569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.203764 (+ 195us) server_negotiation.cc:407] Connection header received
0925 02:32:15.204269 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.204270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.204281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.204335 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:15.205408 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.206258 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.207246 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.207452 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.207534 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.209735 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:15.209736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:15.209737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:32:15.209745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:15.211672 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:15.212124 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:15.212125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:15.212126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:15.212204 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:15.212455 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:15.212456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:15.212456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:15.212627 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:15.212703 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.213009 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.213113 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":23}
I20260925 02:32:15.213261 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.203480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44091 (local address 127.10.13.2:50691)
0925 02:32:15.203658 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.203694 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:15.204177 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:15.204387 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:15.204390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:15.205038 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:15.205315 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.205322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.206338 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.206339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.207148 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.207150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.207554 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.207555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.207677 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.208222 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:15.208231 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:15.209638 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:15.211761 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:32:15.211781 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:32:15.211784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:32:15.212030 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:32:15.212281 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:32:15.212282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:32:15.212282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:32:15.212382 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:32:15.212706 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:15.212711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:15.212826 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 02:32:15.212956 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:32:15.213094 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260925 02:32:15.213806 20901 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44091
I20260925 02:32:15.213917 20901 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:15.214107 20901 heartbeater.cc:511] Master 127.10.13.62:44091 requested a full tablet report, sending...
I20260925 02:32:15.214500 20575 ts_manager.cc:194] Registered new tserver with Master: f797fc06b94b4aa8a543f279db0c778a (127.10.13.2:36687)
I20260925 02:32:15.214864 20575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:50691
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:32:15.232066 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44091
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:32:15.314574 20906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:32:15.314776 20906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:32:15.314800 20906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:32:15.315871 20906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:32:15.315905 20906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:32:15.315932 20906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:32:15.317481 20906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:32:15.317665 20906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:32:15.317836 20906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:32:15.322865 20911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:32:15.322868 20912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:15.322918 20906 server_base.cc:1034] running on GCE node
W20260925 02:32:15.322927 20914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:32:15.323323 20906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:32:15.323566 20906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:32:15.324724 20906 hybrid_clock.cc:648] HybridClock initialized: now 1790303535324701 us; error 36 us; skew 500 ppm
Sep 25 02:32:15 dist-test-slave-q8mg krb5kdc[20531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303535, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:32:15.327288 20906 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:32:15.327666 20906 webserver.cc:460] Webserver started at http://127.10.13.3:41219/ using document root <none> and password file <none>
I20260925 02:32:15.327863 20906 fs_manager.cc:359] Metadata directory not provided
I20260925 02:32:15.327903 20906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:32:15.327997 20906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:32:15.329187 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "51e99d41733844e28c749c9ec0dc79f6"
format_stamp: "Formatted at 2026-09-25 02:32:15 on dist-test-slave-q8mg"
server_key: "a364d1443a7313f49a4a9e0b8dd064a2"
server_key_iv: "664bac64304aa33ff856b91d326eef4b"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.329598 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "51e99d41733844e28c749c9ec0dc79f6"
format_stamp: "Formatted at 2026-09-25 02:32:15 on dist-test-slave-q8mg"
server_key: "a364d1443a7313f49a4a9e0b8dd064a2"
server_key_iv: "664bac64304aa33ff856b91d326eef4b"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.331073 20906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 02:32:15.331777 20921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.331949 20906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:32:15.332052 20906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "51e99d41733844e28c749c9ec0dc79f6"
format_stamp: "Formatted at 2026-09-25 02:32:15 on dist-test-slave-q8mg"
server_key: "a364d1443a7313f49a4a9e0b8dd064a2"
server_key_iv: "664bac64304aa33ff856b91d326eef4b"
server_key_version: "encryptionkey@0"
I20260925 02:32:15.332131 20906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:32:15.340698 20906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:32:15.343053 20906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:32:15.343215 20906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:32:15.343446 20906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:32:15.343755 20906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:32:15.343806 20906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.343847 20906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:32:15.343873 20906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:32:15.349925 20906 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:40457
I20260925 02:32:15.349968 21035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:40457 every 8 connection(s)
I20260925 02:32:15.350297 20906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:32:15 dist-test-slave-q8mg krb5kdc[20531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303535, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:32:15.356860 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 20906
I20260925 02:32:15.356980 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:32:15.357214 10292 external_mini_cluster.cc:1428] Setting key 894efb6e105939deb060b421a7fa4e88
I20260925 02:32:15.360008 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.351007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:49049 (local address 127.10.13.62:44091)
0925 02:32:15.351105 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.351108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.351366 (+ 258us) server_negotiation.cc:407] Connection header received
0925 02:32:15.351929 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.351930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.351943 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.352011 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:15.353014 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.353515 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.354497 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.354687 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.354746 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.357075 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:15.357077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:15.357077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:32:15.357086 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:15.358616 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:15.359044 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:15.359045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:15.359045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:15.359098 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:15.359356 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:15.359357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:15.359357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:15.359496 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:15.359571 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.359879 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.359943 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:32:15.360251 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.351032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:44091 (local address 127.10.13.3:49049)
0925 02:32:15.351211 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.351261 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:15.351828 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:15.352082 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:15.352085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:15.352655 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:32:15.352850 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.352854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.353597 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.353598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.354415 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.354417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.354770 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.354771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.354875 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.355433 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:32:15.355442 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:32:15.356958 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:32:15.358711 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:32:15.358718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:32:15.358724 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:32:15.358952 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:32:15.359157 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:32:15.359158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:32:15.359158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:32:15.359266 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:32:15.359593 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:32:15.359599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:32:15.359735 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 02:32:15.359828 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 02:32:15.359957 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260925 02:32:15.360760 21036 heartbeater.cc:347] Connected to a master server at 127.10.13.62:44091
I20260925 02:32:15.360864 21036 heartbeater.cc:464] Registering TS with master...
I20260925 02:32:15.361078 21036 heartbeater.cc:511] Master 127.10.13.62:44091 requested a full tablet report, sending...
I20260925 02:32:15.361515 20575 ts_manager.cc:194] Registered new tserver with Master: 51e99d41733844e28c749c9ec0dc79f6 (127.10.13.3:40457)
I20260925 02:32:15.361901 20575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:49049
I20260925 02:32:15.369016 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:32:15.376370 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.370379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.10.13.62:44091)
0925 02:32:15.370454 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.370460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.370538 (+ 78us) server_negotiation.cc:407] Connection header received
0925 02:32:15.370590 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.370591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.370600 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.370649 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:32:15.371212 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.371694 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.372407 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.372589 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.372638 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.373927 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:32:15.373929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:32:15.373929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:32:15.373936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:32:15.375305 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:15.375644 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:15.375644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:15.375645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:15.375696 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:32:15.375862 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:32:15.375863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:32:15.375863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:32:15.376005 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:32:15.376089 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.376216 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.376274 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:32:15.378335 20575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:32:15.379926 20575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:32:15.386598 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.382429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:40457 (local address 127.0.0.1:43242)
0925 02:32:15.382588 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.382634 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:15.382791 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:15.383184 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:15.383188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:15.383190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:32:15.383280 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.383282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.384664 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.384665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.385546 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.385549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.386303 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.386304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.386383 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.386398 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:32:15.386443 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 02:32:15.386514 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260925 02:32:15.386970 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.382547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.10.13.3:40457)
0925 02:32:15.382735 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.382905 (+ 170us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.382919 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:32:15.382966 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.382968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.382987 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.383088 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:32:15.383370 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.384535 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.385634 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.386208 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.386724 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.386795 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.386825 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.386864 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260925 02:32:15.387135 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.382667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36687 (local address 127.0.0.1:36080)
0925 02:32:15.383081 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.383091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:15.383143 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:15.383418 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:15.383419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:15.383420 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:32:15.383475 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.383477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.384702 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.384702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.386091 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.386093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.386866 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.386867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.386940 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.386945 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:32:15.386988 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 02:32:15.387035 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":22,"thread_start_us":65,"threads_started":1}
I20260925 02:32:15.387425 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.382840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.10.13.1:42583)
0925 02:32:15.383019 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.383024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.383208 (+ 184us) server_negotiation.cc:407] Connection header received
0925 02:32:15.383332 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.383335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.383351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.383419 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:32:15.383597 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.384425 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.385970 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.386691 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.386752 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.386797 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.387090 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.387131 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260925 02:32:15.387429 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.382881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.10.13.2:36687)
0925 02:32:15.383271 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.383275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.383286 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:32:15.383324 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.383326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.383343 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.383421 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:32:15.383535 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.384603 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.386171 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.386698 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.386775 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.386832 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.387282 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.387332 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":90,"threads_started":1}
I20260925 02:32:15.387588 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.382430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:42583 (local address 127.0.0.1:38202)
0925 02:32:15.383074 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.383085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:15.383230 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:15.383495 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:15.383495 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:15.383497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:32:15.383542 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.383543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.384526 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.384527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.385879 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.385882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.386833 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.386834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.386890 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.386895 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:32:15.386943 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:32:15.386985 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":28,"thread_start_us":66,"threads_started":1}
I20260925 02:32:15.387857 20970 tablet_service.cc:1467] Processing CreateTablet for tablet 04528d9c5988494680e79c10d7105eb8 (DEFAULT_TABLE table=test-table [id=2069bd08d2ed47888b005a7d4896bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:32:15.388090 20835 tablet_service.cc:1467] Processing CreateTablet for tablet 04528d9c5988494680e79c10d7105eb8 (DEFAULT_TABLE table=test-table [id=2069bd08d2ed47888b005a7d4896bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:32:15.388247 20970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04528d9c5988494680e79c10d7105eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:32:15.388402 20835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04528d9c5988494680e79c10d7105eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:32:15.388322 20701 tablet_service.cc:1467] Processing CreateTablet for tablet 04528d9c5988494680e79c10d7105eb8 (DEFAULT_TABLE table=test-table [id=2069bd08d2ed47888b005a7d4896bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:32:15.388666 20701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04528d9c5988494680e79c10d7105eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:32:15.391345 21054 tablet_bootstrap.cc:492] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab: Bootstrap starting.
I20260925 02:32:15.392064 21054 tablet_bootstrap.cc:654] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab: Neither blocks nor log segments found. Creating new log.
I20260925 02:32:15.392491 21054 log.cc:824] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab: Log is configured to *not* fsync() on all Append() calls
I20260925 02:32:15.392765 21055 tablet_bootstrap.cc:492] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: Bootstrap starting.
I20260925 02:32:15.393204 21054 tablet_bootstrap.cc:492] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab: No bootstrap required, opened a new log
I20260925 02:32:15.393262 21054 ts_tablet_manager.cc:1397] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:32:15.393424 21055 tablet_bootstrap.cc:654] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: Neither blocks nor log segments found. Creating new log.
I20260925 02:32:15.393666 21056 tablet_bootstrap.cc:492] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a: Bootstrap starting.
I20260925 02:32:15.394214 21056 tablet_bootstrap.cc:654] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a: Neither blocks nor log segments found. Creating new log.
I20260925 02:32:15.394688 21056 log.cc:824] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:32:15.394853 21054 raft_consensus.cc:348] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.395022 21054 raft_consensus.cc:374] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:32:15.395067 21054 raft_consensus.cc:729] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6f3901ef5f42d0be7afd3e0fed40ab, State: Initialized, Role: FOLLOWER
I20260925 02:32:15.395191 21054 consensus_queue.cc:260] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.395432 21055 log.cc:824] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:32:15.395519 21054 ts_tablet_manager.cc:1428] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:32:15.395558 20767 heartbeater.cc:503] Master 127.10.13.62:44091 was elected leader, sending a full tablet report...
I20260925 02:32:15.396005 21056 tablet_bootstrap.cc:492] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a: No bootstrap required, opened a new log
I20260925 02:32:15.396093 21056 ts_tablet_manager.cc:1397] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:32:15.396293 21055 tablet_bootstrap.cc:492] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: No bootstrap required, opened a new log
I20260925 02:32:15.396358 21055 ts_tablet_manager.cc:1397] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 02:32:15.398125 21056 raft_consensus.cc:348] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.398307 21056 raft_consensus.cc:374] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:32:15.398345 21056 raft_consensus.cc:729] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f797fc06b94b4aa8a543f279db0c778a, State: Initialized, Role: FOLLOWER
I20260925 02:32:15.398290 21055 raft_consensus.cc:348] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.398435 21055 raft_consensus.cc:374] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:32:15.398468 21055 raft_consensus.cc:729] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e99d41733844e28c749c9ec0dc79f6, State: Initialized, Role: FOLLOWER
I20260925 02:32:15.398464 21056 consensus_queue.cc:260] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.398578 21055 consensus_queue.cc:260] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.398762 21056 ts_tablet_manager.cc:1428] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:32:15.398799 20901 heartbeater.cc:503] Master 127.10.13.62:44091 was elected leader, sending a full tablet report...
I20260925 02:32:15.399019 21036 heartbeater.cc:503] Master 127.10.13.62:44091 was elected leader, sending a full tablet report...
I20260925 02:32:15.399334 21055 ts_tablet_manager.cc:1428] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260925 02:32:15.453346 20902 tablet.cc:2369] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:32:15.523130 21062 raft_consensus.cc:482] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:32:15.523260 21062 raft_consensus.cc:504] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.523751 21062 leader_election.cc:288] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f6f3901ef5f42d0be7afd3e0fed40ab (127.10.13.1:42583), f797fc06b94b4aa8a543f279db0c778a (127.10.13.2:36687)
I20260925 02:32:15.526692 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.524057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:39963 (local address 127.10.13.2:36687)
0925 02:32:15.524106 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.524109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.524173 (+ 64us) server_negotiation.cc:407] Connection header received
0925 02:32:15.524252 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.524253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.524265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.524341 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:32:15.524506 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.525036 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.525964 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.526342 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.526422 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.526434 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.526583 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.526622 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:32:15.526692 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.523969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36687 (local address 127.10.13.3:39963)
0925 02:32:15.524085 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.524100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:15.524178 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:15.524346 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:15.524348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:15.524349 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:32:15.524428 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.524429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.525118 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.525120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.525860 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.525862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.526421 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.526422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.526505 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.526509 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:32:15.526577 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:32:15.526620 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 02:32:15.526893 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.524053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:51463 (local address 127.10.13.1:42583)
0925 02:32:15.524128 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.524130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.524357 (+ 227us) server_negotiation.cc:407] Connection header received
0925 02:32:15.524439 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.524440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.524452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.524531 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:32:15.524689 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.525183 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.526140 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.526524 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.526605 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.526617 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.526774 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.526820 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:32:15.526980 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.523969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:42583 (local address 127.10.13.3:51463)
0925 02:32:15.524269 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.524279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:32:15.524372 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:32:15.524540 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:32:15.524541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:32:15.524543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:32:15.524617 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.524619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.525279 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.525280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.526042 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:32:15.526045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.526621 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.526622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:32:15.526698 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.526701 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:32:15.526853 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 02:32:15.526892 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":115,"thread_start_us":124,"threads_started":1}
I20260925 02:32:15.527043 20855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04528d9c5988494680e79c10d7105eb8" candidate_uuid: "51e99d41733844e28c749c9ec0dc79f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f797fc06b94b4aa8a543f279db0c778a" is_pre_election: true
I20260925 02:32:15.527215 20855 raft_consensus.cc:2454] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51e99d41733844e28c749c9ec0dc79f6 in term 0.
I20260925 02:32:15.527328 20721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04528d9c5988494680e79c10d7105eb8" candidate_uuid: "51e99d41733844e28c749c9ec0dc79f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" is_pre_election: true
I20260925 02:32:15.527422 20923 leader_election.cc:302] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51e99d41733844e28c749c9ec0dc79f6, f797fc06b94b4aa8a543f279db0c778a; no voters:
I20260925 02:32:15.527493 20721 raft_consensus.cc:2454] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51e99d41733844e28c749c9ec0dc79f6 in term 0.
I20260925 02:32:15.527580 21062 raft_consensus.cc:2781] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:32:15.527654 21062 raft_consensus.cc:482] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:32:15.527680 21062 raft_consensus.cc:3037] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:32:15.528383 21062 raft_consensus.cc:504] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.528549 21062 leader_election.cc:288] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [CANDIDATE]: Term 1 election: Requested vote from peers 0f6f3901ef5f42d0be7afd3e0fed40ab (127.10.13.1:42583), f797fc06b94b4aa8a543f279db0c778a (127.10.13.2:36687)
I20260925 02:32:15.528678 20721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04528d9c5988494680e79c10d7105eb8" candidate_uuid: "51e99d41733844e28c749c9ec0dc79f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab"
I20260925 02:32:15.528705 20855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04528d9c5988494680e79c10d7105eb8" candidate_uuid: "51e99d41733844e28c749c9ec0dc79f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f797fc06b94b4aa8a543f279db0c778a"
I20260925 02:32:15.528762 20721 raft_consensus.cc:3037] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:32:15.528807 20855 raft_consensus.cc:3037] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:32:15.529596 20721 raft_consensus.cc:2454] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51e99d41733844e28c749c9ec0dc79f6 in term 1.
I20260925 02:32:15.529577 20855 raft_consensus.cc:2454] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51e99d41733844e28c749c9ec0dc79f6 in term 1.
I20260925 02:32:15.529799 20923 leader_election.cc:302] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51e99d41733844e28c749c9ec0dc79f6, f797fc06b94b4aa8a543f279db0c778a; no voters:
I20260925 02:32:15.529913 21062 raft_consensus.cc:2781] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:32:15.530093 21062 raft_consensus.cc:686] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 1 LEADER]: Becoming Leader. State: Replica: 51e99d41733844e28c749c9ec0dc79f6, State: Running, Role: LEADER
I20260925 02:32:15.530197 21062 consensus_queue.cc:237] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } }
I20260925 02:32:15.531062 20576 catalog_manager.cc:5495] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51e99d41733844e28c749c9ec0dc79f6 (127.10.13.3). New cstate: current_term: 1 leader_uuid: "51e99d41733844e28c749c9ec0dc79f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51e99d41733844e28c749c9ec0dc79f6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 } health_report { overall_health: UNKNOWN } } }
W20260925 02:32:15.545094 20768 tablet.cc:2369] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:32:15.568089 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.565424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.10.13.3:40457)
0925 02:32:15.565494 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:32:15.565496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:32:15.565507 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:32:15.565627 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:32:15.565628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:32:15.565652 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:32:15.565735 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:32:15.565875 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.566431 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.567232 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:32:15.567437 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:32:15.567518 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:32:15.567692 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:32:15.567809 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:32:15.567910 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:32:15.567929 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 02:32:15.567951 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:32:15.570004 20855 raft_consensus.cc:1261] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a [term 1 FOLLOWER]: Refusing update from remote peer 51e99d41733844e28c749c9ec0dc79f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:32:15.570008 20721 raft_consensus.cc:1261] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [term 1 FOLLOWER]: Refusing update from remote peer 51e99d41733844e28c749c9ec0dc79f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:32:15.570391 21064 consensus_queue.cc:1035] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f797fc06b94b4aa8a543f279db0c778a" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:32:15.570518 21062 consensus_queue.cc:1035] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6f3901ef5f42d0be7afd3e0fed40ab" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 42583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:32:15.573748 21068 mvcc.cc:205] Tried to move back new op lower bound from 7333083281691852800 to 7333083281532366848. Current Snapshot: MvccSnapshot[applied={T|T < 7333083281691852800}]
I20260925 02:32:15.575376 20575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:32:15.575459 20575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:32:15.576280 20575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c9e932ebeb14d289bd87e83d4e0208f: Sending DeleteTablet for 3 replicas of tablet 04528d9c5988494680e79c10d7105eb8
I20260925 02:32:15.576608 20970 tablet_service.cc:1514] Processing DeleteTablet for tablet 04528d9c5988494680e79c10d7105eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:32:15 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:43242
I20260925 02:32:15.576670 20701 tablet_service.cc:1514] Processing DeleteTablet for tablet 04528d9c5988494680e79c10d7105eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:32:15 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:38202
I20260925 02:32:15.576898 21078 tablet_replica.cc:330] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: stopping tablet replica
I20260925 02:32:15.576858 20835 tablet_service.cc:1514] Processing DeleteTablet for tablet 04528d9c5988494680e79c10d7105eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:32:15 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:36080
I20260925 02:32:15.577090 21080 tablet_replica.cc:330] T 04528d9c5988494680e79c10d7105eb8 P f797fc06b94b4aa8a543f279db0c778a: stopping tablet replica
I20260925 02:32:15.577122 21079 tablet_replica.cc:330] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab: stopping tablet replica
I20260925 02:32:15.577217 21078 raft_consensus.cc:2229] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:32:15.577332 21078 raft_consensus.cc:2258] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:32:15.577353 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 20638
I20260925 02:32:15.577337 21079 raft_consensus.cc:2229] T 04528d9c5988494680e79c10d7105eb8 P 0f6f3901ef5f42d0be7afd3e0fed40ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:32:15.578720 21078 ts_tablet_manager.cc:1905] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:32:15.580753 21078 ts_tablet_manager.cc:1918] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:32:15.580832 21078 log.cc:1196] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/04528d9c5988494680e79c10d7105eb8
I20260925 02:32:15.581162 21078 ts_tablet_manager.cc:1939] T 04528d9c5988494680e79c10d7105eb8 P 51e99d41733844e28c749c9ec0dc79f6: Deleting consensus metadata
I20260925 02:32:15.581635 20561 catalog_manager.cc:4841] TS 51e99d41733844e28c749c9ec0dc79f6 (127.10.13.3:40457): tablet 04528d9c5988494680e79c10d7105eb8 (table test-table [id=2069bd08d2ed47888b005a7d4896bbeb]) successfully deleted
W20260925 02:32:15.584533 20563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:42583 (error 108)
I20260925 02:32:15.584779 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 20772
I20260925 02:32:15.585512 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.584978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:42583 (local address 127.0.0.1:38218)
0925 02:32:15.585425 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.585470 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425}
W20260925 02:32:15.589062 20563 catalog_manager.cc:4584] TS 0f6f3901ef5f42d0be7afd3e0fed40ab (127.10.13.1:42583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04528d9c5988494680e79c10d7105eb8: Network error: Client connection negotiation failed: client connection to 127.10.13.1:42583: connect: Connection refused (error 111)
I20260925 02:32:15.590364 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 20906
I20260925 02:32:15.590394 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:32:15.590285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36687 (local address 127.0.0.1:36096)
0925 02:32:15.590335 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:32:15.590367 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.2:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 02:32:15.595252 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 20544
2026-09-25T02:32:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2049 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:32:15 dist-test-slave-q8mg krb5kdc[21084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:32:15 dist-test-slave-q8mg krb5kdc[21084](info): set up 2 sockets
Sep 25 02:32:15 dist-test-slave-q8mg krb5kdc[21084](info): commencing operation
krb5kdc: starting...
W20260925 02:32:16.724371 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:32:16 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:32:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/rangeradmin_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/rangeradmin_127.10.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/rangerlookup_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/rangerlookup_127.10.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/HTTP_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/HTTP_127.10.13.20@KRBTEST.COM.keytab.
I20260925 02:32:16.900352 10292 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3SrHiO/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/postgres -l logfile start
2026-09-25 02:32:19.588 UTC [21114] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:32:19.588 UTC [21114] LOG: listening on IPv4 address "127.10.13.20", port 37831
2026-09-25 02:32:19.592 UTC [21114] LOG: listening on Unix socket "/tmp/.s.PGSQL.37831"
2026-09-25 02:32:19.628 UTC [21117] LOG: database system was shut down at 2026-09-25 02:32:17 UTC
2026-09-25 02:32:19.633 UTC [21114] LOG: database system is ready to accept connections
I20260925 02:32:20.596429 10292 mini_postgres.cc:96] Postgres bound to 37831
2026-09-25 02:32:20.603 UTC [21125] FATAL: database "slave" does not exist
127.10.13.20:37831 - accepting connections
I20260925 02:32:20.604482 10292 mini_ranger.cc:161] Starting Ranger...
I20260925 02:32:20.621253 10292 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:32:21.153465 10292 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:32:21.153556 10292 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-admin
2026-09-25 02:32:21,486 [I] DB FLAVOR :POSTGRES
2026-09-25 02:32:21,486 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:32:21,486 [I] Checking connection..
2026-09-25 02:32:21,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:32:21,738 [I] Checking connection passed.
2026-09-25 02:32:21,738 [I] --------- Verifying version history table ---------
2026-09-25 02:32:21,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:32:21,980 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:32:21,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:32:22,223 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:32:22,223 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:32:22,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:32:22.461 UTC [21189] WARNING: there is no transaction in progress
2026-09-25 02:32:22,468 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:32:22,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:32:22,747 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:32:22,747 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:32:22,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:32:22,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:32:23,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q8mg.c.gcp-upstream.internal','N') ;"
2026-09-25 02:32:23,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:32:23,755 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:32:23,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:32:24,016 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:32:24,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:32:24,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:32:24,508 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:32:24,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:32:25.512 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.530 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.549 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.566 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.583 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.603 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.670 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.677 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.692 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.704 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.715 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.728 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.739 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.746 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.753 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:25.761 UTC [21298] WARNING: there is no transaction in progress
2026-09-25 02:32:26,477 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:32:26,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:32:26,736 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:32:26,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:32:26,985 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:32:26,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:32:27,233 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:32:27,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:32:27,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:32:27,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q8mg.c.gcp-upstream.internal';"
2026-09-25 02:32:27,956 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:32:27,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:32:28,211 [I] DB_PATCHES have already been applied
I20260925 02:32:28.217737 10292 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-admin:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3SrHiO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:32:28.217815 10292 mini_ranger.cc:193] Using host: 127.10.13.20
I20260925 02:32:28.221222 10292 mini_ranger.cc:227] Ranger admin URL: http://127.10.13.20:50909
Sep 25, 2026 2:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:32:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:32:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:32:29.330081 10292 mini_ranger.cc:160] Time spent starting Ranger: real 8.726s user 0.000s sys 0.009s
Sep 25, 2026 2:32:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:32:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:32:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:32:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:32:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:32:52.183751 10292 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/rangerkms_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/rangerkms_127.10.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.13.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.13.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.13.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/HTTP_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.13.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/HTTP_127.10.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:32:52 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303572, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:32:52.270731 10292 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:32:52.280583 10292 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:32:53.190773 10292 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:32:53.190866 10292 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms
I20260925 02:32:53.190886 10292 mini_ranger_kms.cc:227] Using postgres at 127.10.13.20:37831
2026-09-25 02:32:53,959 [I] DB FLAVOR :POSTGRES
2026-09-25 02:32:53,959 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:32:53,960 [I] Checking connection
2026-09-25 02:32:53,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:32:54,230 [I] connection success
2026-09-25 02:32:54,230 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:32:54,231 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:32:54,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:32:54,464 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:32:54,465 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:32:54,465 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:32:54,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:37831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:32:54,740 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:32:55.267643 10292 mini_ranger_kms.cc:310] Created kms service
I20260925 02:32:55.486832 10292 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:32:55.541576 10292 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:32:55 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303572, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:32:55.974650 10292 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:32:55.974769 10292 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms:/tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:32:55.974793 10292 mini_ranger_kms.cc:240] Using host: 127.10.13.20
I20260925 02:32:55.978885 10292 mini_ranger_kms.cc:276] Ranger KMS PID: 21479
I20260925 02:32:55.979025 10292 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.13.20:52825
Sep 25, 2026 2:32:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:32:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:32:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:32:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:32:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:32:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:32:58.218916 10292 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.948s user 0.023s sys 0.290s
I20260925 02:32:58.219028 10292 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:32:58.219101 10292 mini_ranger_kms.cc:401] 127.10.13.20:52825/kms/v1/keys
Sep 25 02:32:58 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303578, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.13.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:32:58 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303578, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.13.20@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
Sep 25 02:32:59 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303578, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.13.20@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
Sep 25 02:33:01 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:33:01.513110 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:41815
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:41815
--ranger_config_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:52825/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:01.599372 21529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:01.599602 21529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:01.599624 21529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:01.600811 21529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:01.600855 21529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:01.600919 21529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:01.600952 21529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:01.600970 21529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:01.602567 21529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.13.20:52825/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:41815
--ranger_config_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:41815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:01.602828 21529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:01.603094 21529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:01.608253 21535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:01.608249 21537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:01.608263 21534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:01.608671 21529 server_base.cc:1034] running on GCE node
I20260925 02:33:01.609000 21529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:01.609308 21529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:01.610481 21529 hybrid_clock.cc:648] HybridClock initialized: now 1790303581610466 us; error 32 us; skew 500 ppm
Sep 25 02:33:01 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303581, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:01.613211 21529 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:33:01.613758 21529 webserver.cc:460] Webserver started at http://127.10.13.62:38359/ using document root <none> and password file <none>
I20260925 02:33:01.614017 21529 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:01.614078 21529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:01.614240 21529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:33:01 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303581, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:01.678478 21529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "ef37998390af41e1a9183f55aac99852"
format_stamp: "Formatted at 2026-09-25 02:33:01 on dist-test-slave-q8mg"
server_key: "eec8961c1628c0954a6eccfdafdf2c39"
server_key_iv: "61b958e766b499202c93219556a473e7"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:01.679083 21529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "ef37998390af41e1a9183f55aac99852"
format_stamp: "Formatted at 2026-09-25 02:33:01 on dist-test-slave-q8mg"
server_key: "eec8961c1628c0954a6eccfdafdf2c39"
server_key_iv: "61b958e766b499202c93219556a473e7"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:01.680989 21529 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:33:01.697636 21544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:01.697907 21529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:33:01.698032 21529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "ef37998390af41e1a9183f55aac99852"
format_stamp: "Formatted at 2026-09-25 02:33:01 on dist-test-slave-q8mg"
server_key: "eec8961c1628c0954a6eccfdafdf2c39"
server_key_iv: "61b958e766b499202c93219556a473e7"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:01.698136 21529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:01.714231 21529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:01.716624 21529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:01.716781 21529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:01.723222 21529 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:41815
I20260925 02:33:01.723320 21596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:41815 every 8 connection(s)
I20260925 02:33:01.723695 21529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:33:01.724429 21597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:01.726907 21597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852: Bootstrap starting.
I20260925 02:33:01.727535 21597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:01.728083 21597 log.cc:824] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:01.728781 21597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852: No bootstrap required, opened a new log
I20260925 02:33:01.729987 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 21529
I20260925 02:33:01.730144 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:33:01.730346 10292 external_mini_cluster.cc:1428] Setting key c4e2bc363c02eabf6044e6d785f50613
I20260925 02:33:01.731688 21597 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef37998390af41e1a9183f55aac99852" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41815 } }
I20260925 02:33:01.731851 21597 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:01.731921 21597 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef37998390af41e1a9183f55aac99852, State: Initialized, Role: FOLLOWER
I20260925 02:33:01.732126 21597 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef37998390af41e1a9183f55aac99852" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41815 } }
I20260925 02:33:01.732225 21597 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:33:01.732259 21597 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:33:01.732342 21597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:01.733184 21597 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef37998390af41e1a9183f55aac99852" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41815 } }
I20260925 02:33:01.733319 21597 leader_election.cc:302] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef37998390af41e1a9183f55aac99852; no voters:
I20260925 02:33:01.733507 21597 leader_election.cc:288] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:33:01.733719 21603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:33:01.733795 21597 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:33:01.733927 21603 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [term 1 LEADER]: Becoming Leader. State: Replica: ef37998390af41e1a9183f55aac99852, State: Running, Role: LEADER
I20260925 02:33:01.734112 21603 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef37998390af41e1a9183f55aac99852" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41815 } }
I20260925 02:33:01.734529 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef37998390af41e1a9183f55aac99852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef37998390af41e1a9183f55aac99852" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41815 } } }
I20260925 02:33:01.734679 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:01.734843 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef37998390af41e1a9183f55aac99852. Latest consensus state: current_term: 1 leader_uuid: "ef37998390af41e1a9183f55aac99852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef37998390af41e1a9183f55aac99852" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41815 } } }
I20260925 02:33:01.734932 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:01.735073 21611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:33:01.735383 21597 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:33:01.735610 21611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:33:01 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:01.746730 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:01.730916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.10.13.62:41815)
0925 02:33:01.731114 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:01.731125 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:01.731161 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:33:01.731688 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:01.731699 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:01.731767 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:01.731915 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:01.732855 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:01.734413 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:01.735565 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:01.738164 (+ 2599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:01.738406 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:01.741709 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:01.741714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:01.741714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:01.741731 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:01.744397 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:01.744918 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:01.744922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:01.744922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:01.745058 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:01.745321 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:01.745322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:01.745323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:01.745693 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:01.746264 (+ 571us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:01.746339 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:33:01.746482 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260925 02:33:01.747053 21611 catalog_manager.cc:1347] Generated new cluster ID: e5fa535a2ae0412abcdeb2360de6cd19
I20260925 02:33:01.747109 21611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:33:01.781020 21611 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:33:01.781873 21611 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:33:01.794162 21611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852: Generated new TSK 0
I20260925 02:33:01.794509 21611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:33:02.552316 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:52825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41815
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:02.651232 21642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:02.651487 21642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:02.651543 21642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:02.652791 21642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:02.652858 21642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:02.652917 21642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:33:02.654469 21642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.13.20:52825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:02.654742 21642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:02.654989 21642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:02.660475 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:02.660672 21642 server_base.cc:1034] running on GCE node
W20260925 02:33:02.660815 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:02.660933 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:02.661190 21642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:02.661506 21642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:02.662698 21642 hybrid_clock.cc:648] HybridClock initialized: now 1790303582662679 us; error 36 us; skew 500 ppm
Sep 25 02:33:02 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303582, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:02.665761 21642 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:33:02.666249 21642 webserver.cc:460] Webserver started at http://127.10.13.1:37061/ using document root <none> and password file <none>
I20260925 02:33:02.666491 21642 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:02.666561 21642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:02.666700 21642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:33:02 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303582, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:02.680949 21642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "3956bfa61c5a474eb65197e156d3a2dd"
format_stamp: "Formatted at 2026-09-25 02:33:02 on dist-test-slave-q8mg"
server_key: "5dd269a2d594d310033f2e440b7558fd"
server_key_iv: "961cefa0bc5239d4ea57ca87b5da5459"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:02.681523 21642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "3956bfa61c5a474eb65197e156d3a2dd"
format_stamp: "Formatted at 2026-09-25 02:33:02 on dist-test-slave-q8mg"
server_key: "5dd269a2d594d310033f2e440b7558fd"
server_key_iv: "961cefa0bc5239d4ea57ca87b5da5459"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:02.689206 21642 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.002s
I20260925 02:33:02.701947 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:02.702167 21642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:33:02.702292 21642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "3956bfa61c5a474eb65197e156d3a2dd"
format_stamp: "Formatted at 2026-09-25 02:33:02 on dist-test-slave-q8mg"
server_key: "5dd269a2d594d310033f2e440b7558fd"
server_key_iv: "961cefa0bc5239d4ea57ca87b5da5459"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:02.702399 21642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:02.720741 21642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:02.722954 21642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:02.723172 21642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:02.723472 21642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:02.723865 21642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:02.723932 21642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:02.723980 21642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:02.724013 21642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:02.733316 21642 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:37957
I20260925 02:33:02.733507 21773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:37957 every 8 connection(s)
I20260925 02:33:02.733821 21642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:33:02.739356 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 21642
I20260925 02:33:02.739473 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:33:02.739667 10292 external_mini_cluster.cc:1428] Setting key 77f84388ffbef93a2915046e215f72d7
Sep 25 02:33:02 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303582, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:02.746284 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:02.734880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:42523 (local address 127.10.13.62:41815)
0925 02:33:02.735022 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:02.735025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:02.735236 (+ 211us) server_negotiation.cc:407] Connection header received
0925 02:33:02.735957 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:02.735959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:02.735974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:02.736046 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:02.736894 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:02.737729 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:02.739433 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:02.739741 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:02.739789 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:02.742676 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:02.742677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:02.742678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:02.742686 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:02.744493 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:02.745074 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:02.745075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:02.745076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:02.745142 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:02.745430 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:02.745431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:02.745431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:02.745594 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:02.745646 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:02.746067 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 02:33:02.746165 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260925 02:33:02.746994 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:02.734883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41815 (local address 127.10.13.1:42523)
0925 02:33:02.735095 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:02.735138 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:02.735832 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:02.736128 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:02.736133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:02.736434 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:02.736757 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:02.736762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:02.738022 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:02.738023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:02.739322 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:02.739325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:02.739893 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:02.739893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:02.740044 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:02.740753 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:02.740767 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:02.742542 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:02.744615 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:02.744622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:02.744624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:02.744974 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:02.745216 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:02.745217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:02.745217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:02.745352 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:02.745699 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:02.745705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:02.745892 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 02:33:02.746568 (+ 676us) client_negotiation.cc:240] Negotiation successful
0925 02:33:02.746747 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260925 02:33:02.747629 21774 heartbeater.cc:347] Connected to a master server at 127.10.13.62:41815
I20260925 02:33:02.747730 21774 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:02.747936 21774 heartbeater.cc:511] Master 127.10.13.62:41815 requested a full tablet report, sending...
I20260925 02:33:02.748585 21561 ts_manager.cc:194] Registered new tserver with Master: 3956bfa61c5a474eb65197e156d3a2dd (127.10.13.1:37957)
I20260925 02:33:02.749563 21561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:42523
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:33:02.776497 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:52825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41815
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:02.907833 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:02.908234 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:02.908334 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:02.910482 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:02.910640 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:02.910737 21782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:33:02.913355 21782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.13.20:52825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:02.913930 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:02.914340 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:02.922202 21792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:02.922236 21790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:02.923326 21789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:02.923641 21782 server_base.cc:1034] running on GCE node
I20260925 02:33:02.924015 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:02.924333 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:02.926024 21782 hybrid_clock.cc:648] HybridClock initialized: now 1790303582925985 us; error 38 us; skew 500 ppm
Sep 25 02:33:02 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303582, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:02.931025 21782 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:33:02.931677 21782 webserver.cc:460] Webserver started at http://127.10.13.2:36257/ using document root <none> and password file <none>
I20260925 02:33:02.932024 21782 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:02.932157 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:02.932374 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:33:02 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303582, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:02.952595 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "c0d7a2f036504134b651769bf34112fa"
format_stamp: "Formatted at 2026-09-25 02:33:02 on dist-test-slave-q8mg"
server_key: "e1921e1c8a0e51a85cfcb73e333aabdd"
server_key_iv: "f8d606b0968ccb2d0a0d745554a28621"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:02.953202 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "c0d7a2f036504134b651769bf34112fa"
format_stamp: "Formatted at 2026-09-25 02:33:02 on dist-test-slave-q8mg"
server_key: "e1921e1c8a0e51a85cfcb73e333aabdd"
server_key_iv: "f8d606b0968ccb2d0a0d745554a28621"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:02.957820 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 02:33:02.974659 21799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:02.974938 21782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:02.975102 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "c0d7a2f036504134b651769bf34112fa"
format_stamp: "Formatted at 2026-09-25 02:33:02 on dist-test-slave-q8mg"
server_key: "e1921e1c8a0e51a85cfcb73e333aabdd"
server_key_iv: "f8d606b0968ccb2d0a0d745554a28621"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:02.975281 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:02.996101 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 25 02:33:02 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303582, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:02.998917 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:02.999140 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:02.999416 21782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:02.999859 21782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:02.999953 21782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:03.000034 21782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:03.000098 21782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:03.019764 21782 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:44819
I20260925 02:33:03.020372 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:33:03.020946 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 21782
I20260925 02:33:03.021067 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:33:03.021283 10292 external_mini_cluster.cc:1428] Setting key cbb83436a0247b8276d69d14191081f7
I20260925 02:33:03.025472 21912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:44819 every 8 connection(s)
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Sep 25 02:33:03 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303582, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:03.044739 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.023064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:57825 (local address 127.10.13.62:41815)
0925 02:33:03.023108 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.023110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.030676 (+ 7566us) server_negotiation.cc:407] Connection header received
0925 02:33:03.031596 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.031598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.031612 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.031645 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:03.035771 (+ 4126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.036605 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.038237 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.038571 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.038619 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.041405 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:03.041406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:03.041407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:03.041415 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:03.043108 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:03.043609 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:03.043611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:03.043611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:03.043674 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:03.043945 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:03.043946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:03.043946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:03.044084 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:03.044137 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.044524 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.044616 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:33:03.045346 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.022773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41815 (local address 127.10.13.2:57825)
0925 02:33:03.030503 (+ 7730us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.030562 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:03.031443 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:03.031848 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:03.031853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:03.032132 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:03.032430 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.032437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.036750 (+ 4313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.036751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.038128 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.038132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.038725 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.038726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.038877 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.039601 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:03.039612 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:03.041286 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:03.043205 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:03.043213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:03.043216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:03.043520 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:03.043740 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:03.043740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:03.043740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:03.043873 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:03.044171 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:03.044177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:03.044362 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 02:33:03.044920 (+ 558us) client_negotiation.cc:240] Negotiation successful
0925 02:33:03.045100 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7665,"thread_start_us":94,"threads_started":1}
I20260925 02:33:03.045989 21913 heartbeater.cc:347] Connected to a master server at 127.10.13.62:41815
I20260925 02:33:03.046093 21913 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:03.046286 21913 heartbeater.cc:511] Master 127.10.13.62:41815 requested a full tablet report, sending...
I20260925 02:33:03.046684 21561 ts_manager.cc:194] Registered new tserver with Master: c0d7a2f036504134b651769bf34112fa (127.10.13.2:44819)
I20260925 02:33:03.047268 21561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:57825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:33:03.068369 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:52825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41815
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:03.188238 21918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:03.188509 21918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:03.188555 21918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:03.189903 21918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:03.189977 21918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:03.190023 21918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:33:03.191656 21918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.13.20:52825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:41815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:03.191923 21918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:03.192185 21918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:03.198662 21924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:03.198362 21927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:03.198563 21918 server_base.cc:1034] running on GCE node
W20260925 02:33:03.198477 21925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:03.199252 21918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:03.199597 21918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:03.200796 21918 hybrid_clock.cc:648] HybridClock initialized: now 1790303583200774 us; error 48 us; skew 500 ppm
Sep 25 02:33:03 dist-test-slave-q8mg krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303583, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:03.204351 21918 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:33:03.204921 21918 webserver.cc:460] Webserver started at http://127.10.13.3:41395/ using document root <none> and password file <none>
I20260925 02:33:03.205206 21918 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:03.205278 21918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:03.205439 21918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:33:03 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303583, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:03.221421 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "ee373561c0244b4e97c1f13566fec8ba"
format_stamp: "Formatted at 2026-09-25 02:33:03 on dist-test-slave-q8mg"
server_key: "074cb700ba23a7f3e115e6421c3471d8"
server_key_iv: "cf9de4aabdc023e7ab641c262889f191"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:03.221892 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "ee373561c0244b4e97c1f13566fec8ba"
format_stamp: "Formatted at 2026-09-25 02:33:03 on dist-test-slave-q8mg"
server_key: "074cb700ba23a7f3e115e6421c3471d8"
server_key_iv: "cf9de4aabdc023e7ab641c262889f191"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:03.227289 21918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260925 02:33:03.240000 21934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:03.240190 21918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:03.240281 21918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "ee373561c0244b4e97c1f13566fec8ba"
format_stamp: "Formatted at 2026-09-25 02:33:03 on dist-test-slave-q8mg"
server_key: "074cb700ba23a7f3e115e6421c3471d8"
server_key_iv: "cf9de4aabdc023e7ab641c262889f191"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:03.240376 21918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:03.265949 21918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:03.268936 21918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:03.269109 21918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:03.269354 21918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:03.269770 21918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:03.269840 21918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:03.270412 21918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:03.270459 21918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:03.288193 21918 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:43545
I20260925 02:33:03.288683 21918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:33:03.291680 22047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:43545 every 8 connection(s)
I20260925 02:33:03.297803 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 21918
I20260925 02:33:03.297928 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:33:03.298122 10292 external_mini_cluster.cc:1428] Setting key 2d669d2a90098dd9cb3fcc68361e5bf2
Sep 25 02:33:03 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303583, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:03.306202 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.292358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:42917 (local address 127.10.13.62:41815)
0925 02:33:03.292401 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.292403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.292413 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:33:03.293037 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.293039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.293052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.293082 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:03.293982 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.294765 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.299772 (+ 5007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.300087 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.300158 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.302786 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:03.302788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:03.302788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:03.302797 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:03.304566 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:03.305115 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:03.305116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:03.305117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:03.305175 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:03.305448 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:03.305449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:03.305449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:03.305598 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:03.305650 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.306017 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.306101 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:33:03.306723 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.291948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41815 (local address 127.10.13.3:42917)
0925 02:33:03.292161 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.292209 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:03.292902 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:03.293276 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:03.293280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:03.293550 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:03.293871 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.293877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.295170 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.295171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.299664 (+ 4493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.299668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.300229 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.300230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.300388 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.301015 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:03.301027 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:03.302675 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:03.304685 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:03.304693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:03.304695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:03.305028 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:03.305244 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:03.305244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:03.305245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:03.305376 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:03.305683 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:03.305690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:03.305866 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 02:33:03.306311 (+ 445us) client_negotiation.cc:240] Negotiation successful
0925 02:33:03.306475 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":131,"threads_started":1}
I20260925 02:33:03.307142 22048 heartbeater.cc:347] Connected to a master server at 127.10.13.62:41815
I20260925 02:33:03.307243 22048 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:03.307436 22048 heartbeater.cc:511] Master 127.10.13.62:41815 requested a full tablet report, sending...
I20260925 02:33:03.307794 21561 ts_manager.cc:194] Registered new tserver with Master: ee373561c0244b4e97c1f13566fec8ba (127.10.13.3:43545)
I20260925 02:33:03.308274 21561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:42917
I20260925 02:33:03.309695 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:33:03.319360 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.311364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.10.13.62:41815)
0925 02:33:03.311411 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.311413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.311708 (+ 295us) server_negotiation.cc:407] Connection header received
0925 02:33:03.311799 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.311800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.311810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.311838 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:03.312591 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.313359 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.315090 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.315292 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.315402 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.316706 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:03.316707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:03.316708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:03.316715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:03.318209 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:03.318582 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:03.318583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:03.318584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:03.318643 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:03.318846 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:03.318847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:03.318847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:03.318968 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:03.319076 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.319183 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.319256 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:33:03.321260 21561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:33:03.323539 21561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:33:03.336529 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.326396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:43545 (local address 127.0.0.1:39490)
0925 02:33:03.331345 (+ 4949us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.331356 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:03.331456 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:03.331796 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:03.331800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:03.331802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:03.331886 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.331888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.333266 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.333267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.335059 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.335061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.336199 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.336200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.336274 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.336289 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:33:03.336378 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:33:03.336439 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4929,"thread_start_us":67,"threads_started":1}
I20260925 02:33:03.337177 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.326701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.10.13.3:43545)
0925 02:33:03.326910 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.326916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.331421 (+ 4505us) server_negotiation.cc:407] Connection header received
0925 02:33:03.331572 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.331575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.331597 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.331681 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:03.331977 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.333169 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.335169 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.336011 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.336084 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.336166 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.337048 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.337090 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260925 02:33:03.338382 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 43d9e326cf66443fb824f5268f2cc509 (DEFAULT_TABLE table=test-table [id=6d6b6f9d26f54a10ab0801bc0e4b1999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:03.338794 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d9e326cf66443fb824f5268f2cc509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:03.339191 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.326154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:44819 (local address 127.0.0.1:39982)
0925 02:33:03.329243 (+ 3089us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.329250 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:03.329275 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:03.333349 (+ 4074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:03.333350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:03.333352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:03.333401 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.333402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.337003 (+ 3601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.337004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.337969 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.337971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.338941 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.338942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.339014 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.339018 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:33:03.339067 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:33:03.339109 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3077,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260925 02:33:03.339210 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.326154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37957 (local address 127.0.0.1:52434)
0925 02:33:03.329038 (+ 2884us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.329088 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:03.329159 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:03.332028 (+ 2869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:03.332030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:03.332032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:03.332066 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.332068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.334908 (+ 2840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.334909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.336009 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.336011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.339028 (+ 3017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.339029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.339088 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.339091 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:33:03.339118 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 02:33:03.339148 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2849,"thread_start_us":90,"threads_started":1}
I20260925 02:33:03.339741 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.326591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.10.13.2:44819)
0925 02:33:03.331636 (+ 5045us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.331642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.331652 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:33:03.331690 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.331703 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.331717 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.333512 (+ 1795us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:03.333590 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.334799 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.338068 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.338782 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.338873 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.338962 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.339607 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.339656 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4998,"thread_start_us":84,"threads_started":1}
I20260925 02:33:03.339771 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.326432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.10.13.1:37957)
0925 02:33:03.331850 (+ 5418us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.331854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.331864 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:33:03.331895 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.331909 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.331927 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.332138 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:03.332200 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.333250 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.336128 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.336886 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.339288 (+ 2402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.339367 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.339393 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.339433 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5381,"thread_start_us":61,"threads_started":1}
I20260925 02:33:03.340365 21838 tablet_service.cc:1467] Processing CreateTablet for tablet 43d9e326cf66443fb824f5268f2cc509 (DEFAULT_TABLE table=test-table [id=6d6b6f9d26f54a10ab0801bc0e4b1999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:03.340699 21838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d9e326cf66443fb824f5268f2cc509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:03.341580 21706 tablet_service.cc:1467] Processing CreateTablet for tablet 43d9e326cf66443fb824f5268f2cc509 (DEFAULT_TABLE table=test-table [id=6d6b6f9d26f54a10ab0801bc0e4b1999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:03.341914 21706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d9e326cf66443fb824f5268f2cc509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:03.344290 22067 tablet_bootstrap.cc:492] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa: Bootstrap starting.
I20260925 02:33:03.346161 22067 tablet_bootstrap.cc:654] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:03.346484 22067 log.cc:824] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:03.346917 22066 tablet_bootstrap.cc:492] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Bootstrap starting.
I20260925 02:33:03.347381 22067 tablet_bootstrap.cc:492] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa: No bootstrap required, opened a new log
I20260925 02:33:03.347569 22067 ts_tablet_manager.cc:1397] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260925 02:33:03.348037 22066 tablet_bootstrap.cc:654] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:03.348592 22066 log.cc:824] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:03.349076 22068 tablet_bootstrap.cc:492] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd: Bootstrap starting.
I20260925 02:33:03.350116 22068 tablet_bootstrap.cc:654] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:03.350469 22067 raft_consensus.cc:348] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.350809 22067 raft_consensus.cc:374] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:03.350835 22068 log.cc:824] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:03.350927 22067 raft_consensus.cc:729] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d7a2f036504134b651769bf34112fa, State: Initialized, Role: FOLLOWER
I20260925 02:33:03.351128 22067 consensus_queue.cc:260] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.351527 22067 ts_tablet_manager.cc:1428] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:33:03.351626 22066 tablet_bootstrap.cc:492] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: No bootstrap required, opened a new log
I20260925 02:33:03.351699 22066 ts_tablet_manager.cc:1397] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 02:33:03.353005 21913 heartbeater.cc:503] Master 127.10.13.62:41815 was elected leader, sending a full tablet report...
I20260925 02:33:03.354030 22066 raft_consensus.cc:348] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.354198 22066 raft_consensus.cc:374] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:03.354241 22066 raft_consensus.cc:729] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee373561c0244b4e97c1f13566fec8ba, State: Initialized, Role: FOLLOWER
I20260925 02:33:03.354374 22066 consensus_queue.cc:260] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.354755 22066 ts_tablet_manager.cc:1428] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:33:03.354807 22048 heartbeater.cc:503] Master 127.10.13.62:41815 was elected leader, sending a full tablet report...
I20260925 02:33:03.355736 22068 tablet_bootstrap.cc:492] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd: No bootstrap required, opened a new log
I20260925 02:33:03.355816 22068 ts_tablet_manager.cc:1397] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 02:33:03.369509 22068 raft_consensus.cc:348] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.369777 22068 raft_consensus.cc:374] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:03.369834 22068 raft_consensus.cc:729] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3956bfa61c5a474eb65197e156d3a2dd, State: Initialized, Role: FOLLOWER
I20260925 02:33:03.370026 22068 consensus_queue.cc:260] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.370369 22068 ts_tablet_manager.cc:1428] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260925 02:33:03.370512 21774 heartbeater.cc:503] Master 127.10.13.62:41815 was elected leader, sending a full tablet report...
W20260925 02:33:03.485193 21775 tablet.cc:2369] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:33:03 dist-test-slave-q8mg krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303582, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
W20260925 02:33:03.530802 21914 tablet.cc:2369] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:33:03.541455 22049 tablet.cc:2369] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:33:03.573856 22074 raft_consensus.cc:482] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:33:03.574020 22074 raft_consensus.cc:504] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.574551 22074 leader_election.cc:288] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0d7a2f036504134b651769bf34112fa (127.10.13.2:44819), ee373561c0244b4e97c1f13566fec8ba (127.10.13.3:43545)
I20260925 02:33:03.578926 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.575156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:35065 (local address 127.10.13.2:44819)
0925 02:33:03.575226 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.575230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.575538 (+ 308us) server_negotiation.cc:407] Connection header received
0925 02:33:03.575644 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.575645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.575655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.575686 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:03.575922 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.576633 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.578056 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.578558 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.578603 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.578615 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.578815 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.578853 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:33:03.579409 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.574844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:44819 (local address 127.10.13.1:35065)
0925 02:33:03.575466 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.575479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:03.575570 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:03.575757 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:03.575758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:03.575769 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:03.575848 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.575850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.576737 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.576738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.577946 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.577949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.578671 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.578672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.578738 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.578742 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:33:03.579273 (+ 531us) client_negotiation.cc:240] Negotiation successful
0925 02:33:03.579319 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":110,"threads_started":1}
I20260925 02:33:03.579933 21867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d9e326cf66443fb824f5268f2cc509" candidate_uuid: "3956bfa61c5a474eb65197e156d3a2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d7a2f036504134b651769bf34112fa" is_pre_election: true
I20260925 02:33:03.580143 21867 raft_consensus.cc:2454] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3956bfa61c5a474eb65197e156d3a2dd in term 0.
I20260925 02:33:03.580441 21661 leader_election.cc:302] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3956bfa61c5a474eb65197e156d3a2dd, c0d7a2f036504134b651769bf34112fa; no voters:
I20260925 02:33:03.580595 22074 raft_consensus.cc:2781] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:33:03.580637 22074 raft_consensus.cc:482] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:33:03.580658 22074 raft_consensus.cc:3037] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:03.581736 22074 raft_consensus.cc:504] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.581887 22074 leader_election.cc:288] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [CANDIDATE]: Term 1 election: Requested vote from peers c0d7a2f036504134b651769bf34112fa (127.10.13.2:44819), ee373561c0244b4e97c1f13566fec8ba (127.10.13.3:43545)
I20260925 02:33:03.582165 21867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d9e326cf66443fb824f5268f2cc509" candidate_uuid: "3956bfa61c5a474eb65197e156d3a2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d7a2f036504134b651769bf34112fa"
I20260925 02:33:03.582242 21867 raft_consensus.cc:3037] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:03.583046 21867 raft_consensus.cc:2454] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3956bfa61c5a474eb65197e156d3a2dd in term 1.
I20260925 02:33:03.583276 21661 leader_election.cc:302] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3956bfa61c5a474eb65197e156d3a2dd, c0d7a2f036504134b651769bf34112fa; no voters:
I20260925 02:33:03.583398 22074 raft_consensus.cc:2781] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:33:03.583554 22074 raft_consensus.cc:686] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 1 LEADER]: Becoming Leader. State: Replica: 3956bfa61c5a474eb65197e156d3a2dd, State: Running, Role: LEADER
I20260925 02:33:03.583663 22074 consensus_queue.cc:237] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } }
I20260925 02:33:03.584555 21561 catalog_manager.cc:5495] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3956bfa61c5a474eb65197e156d3a2dd (127.10.13.1). New cstate: current_term: 1 leader_uuid: "3956bfa61c5a474eb65197e156d3a2dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3956bfa61c5a474eb65197e156d3a2dd" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 } health_report { overall_health: UNKNOWN } } }
I20260925 02:33:03.588025 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.575376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:38641 (local address 127.10.13.3:43545)
0925 02:33:03.575424 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.575426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.579843 (+ 4417us) server_negotiation.cc:407] Connection header received
0925 02:33:03.579941 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.579942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.579955 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.579984 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:03.585071 (+ 5087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.585799 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.587088 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.587652 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.587703 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.587715 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.587913 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.587954 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:33:03.588510 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.575297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:43545 (local address 127.10.13.1:38641)
0925 02:33:03.579791 (+ 4494us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.579802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:03.579873 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:03.580043 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:03.580043 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:03.580045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:03.580158 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.580159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.585930 (+ 5771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.585931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.586976 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:03.586980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.587774 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.587775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:03.587833 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.587836 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:33:03.588333 (+ 497us) client_negotiation.cc:240] Negotiation successful
0925 02:33:03.588411 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4481,"thread_start_us":91,"threads_started":1}
I20260925 02:33:03.589149 22002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d9e326cf66443fb824f5268f2cc509" candidate_uuid: "3956bfa61c5a474eb65197e156d3a2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee373561c0244b4e97c1f13566fec8ba" is_pre_election: true
I20260925 02:33:03.589330 22002 raft_consensus.cc:2454] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3956bfa61c5a474eb65197e156d3a2dd in term 0.
I20260925 02:33:03.589555 22001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d9e326cf66443fb824f5268f2cc509" candidate_uuid: "3956bfa61c5a474eb65197e156d3a2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee373561c0244b4e97c1f13566fec8ba"
I20260925 02:33:03.589709 22001 raft_consensus.cc:3037] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:03.590843 22001 raft_consensus.cc:2454] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3956bfa61c5a474eb65197e156d3a2dd in term 1.
I20260925 02:33:03.616202 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.613535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.10.13.1:37957)
0925 02:33:03.613601 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:03.613603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:03.613611 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:33:03.613658 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:03.613659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:03.613679 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:03.613757 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:33:03.614005 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.614547 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.615320 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:03.615589 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:03.615662 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:03.615842 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:33:03.615972 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:33:03.616063 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:03.616086 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 02:33:03.616108 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:33:03.618194 22001 raft_consensus.cc:1261] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 1 FOLLOWER]: Refusing update from remote peer 3956bfa61c5a474eb65197e156d3a2dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:33:03.618228 21867 raft_consensus.cc:1261] T 43d9e326cf66443fb824f5268f2cc509 P c0d7a2f036504134b651769bf34112fa [term 1 FOLLOWER]: Refusing update from remote peer 3956bfa61c5a474eb65197e156d3a2dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:33:03.618575 22074 consensus_queue.cc:1035] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d7a2f036504134b651769bf34112fa" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:33:03.618765 22074 consensus_queue.cc:1035] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee373561c0244b4e97c1f13566fec8ba" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:33:03.621594 22080 mvcc.cc:205] Tried to move back new op lower bound from 7333083478497415168 to 7333083478359142400. Current Snapshot: MvccSnapshot[applied={T|T < 7333083478497415168}]
I20260925 02:33:03.622123 22081 mvcc.cc:205] Tried to move back new op lower bound from 7333083478497415168 to 7333083478359142400. Current Snapshot: MvccSnapshot[applied={T|T < 7333083478497415168}]
I20260925 02:33:03.624051 22082 mvcc.cc:205] Tried to move back new op lower bound from 7333083478497415168 to 7333083478359142400. Current Snapshot: MvccSnapshot[applied={T|T < 7333083478497415168 or (T in {7333083478497415168})}]
I20260925 02:33:03.626541 21560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:33:03.626633 21560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:33:03.627355 21560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef37998390af41e1a9183f55aac99852: Sending DeleteTablet for 3 replicas of tablet 43d9e326cf66443fb824f5268f2cc509
I20260925 02:33:03.627712 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d9e326cf66443fb824f5268f2cc509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:03 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39490
I20260925 02:33:03.627702 21838 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d9e326cf66443fb824f5268f2cc509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:03 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:39982
I20260925 02:33:03.628012 21706 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d9e326cf66443fb824f5268f2cc509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:03 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:52434
I20260925 02:33:03.628209 22092 tablet_replica.cc:330] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd: stopping tablet replica
I20260925 02:33:03.628297 22092 raft_consensus.cc:2229] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:33:03.628456 22092 raft_consensus.cc:2258] T 43d9e326cf66443fb824f5268f2cc509 P 3956bfa61c5a474eb65197e156d3a2dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:33:03.628476 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 21642
I20260925 02:33:03.630255 22091 tablet_replica.cc:330] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: stopping tablet replica
I20260925 02:33:03.630406 22091 raft_consensus.cc:2229] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:33:03.630534 22091 raft_consensus.cc:2258] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:33:03.631220 22091 ts_tablet_manager.cc:1905] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:33:03.633476 22091 ts_tablet_manager.cc:1918] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:33:03.633579 22091 log.cc:1196] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/43d9e326cf66443fb824f5268f2cc509
I20260925 02:33:03.633872 22091 ts_tablet_manager.cc:1939] T 43d9e326cf66443fb824f5268f2cc509 P ee373561c0244b4e97c1f13566fec8ba: Deleting consensus metadata
I20260925 02:33:03.634660 21548 catalog_manager.cc:4841] TS ee373561c0244b4e97c1f13566fec8ba (127.10.13.3:43545): tablet 43d9e326cf66443fb824f5268f2cc509 (table test-table [id=6d6b6f9d26f54a10ab0801bc0e4b1999]) successfully deleted
W20260925 02:33:03.637272 21545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:37957 (error 108)
I20260925 02:33:03.638105 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 21782
I20260925 02:33:03.638265 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.638137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37957 (local address 127.0.0.1:52442)
0925 02:33:03.638196 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.638232 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 02:33:03.639045 21545 catalog_manager.cc:4584] TS 3956bfa61c5a474eb65197e156d3a2dd (127.10.13.1:37957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43d9e326cf66443fb824f5268f2cc509: Network error: Client connection negotiation failed: client connection to 127.10.13.1:37957: connect: Connection refused (error 111)
I20260925 02:33:03.645702 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 21918
I20260925 02:33:03.646701 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:03.646593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:44819 (local address 127.0.0.1:39984)
0925 02:33:03.646643 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:03.646667 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.2:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:33:03.651710 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 21529
I20260925 02:33:03.658814 10292 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:33:04.003659 10292 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:33:04.003816 10292 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:33:04.348739 10292 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:33:04.349 UTC [21114] LOG: received smart shutdown request
2026-09-25 02:33:04.353 UTC [21114] LOG: background worker "logical replication launcher" (PID 21123) exited with exit code 1
2026-09-25 02:33:04.353 UTC [21118] LOG: shutting down
2026-09-25 02:33:04.561 UTC [21114] LOG: database system is shut down
2026-09-25T02:33:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49079 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:33:04 dist-test-slave-q8mg krb5kdc[22102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:33:04 dist-test-slave-q8mg krb5kdc[22102](info): set up 2 sockets
Sep 25 02:33:04 dist-test-slave-q8mg krb5kdc[22102](info): commencing operation
krb5kdc: starting...
W20260925 02:33:05.727293 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:33:05 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:33:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:33:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/rangeradmin_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/rangeradmin_127.10.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/rangerlookup_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/rangerlookup_127.10.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/HTTP_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/HTTP_127.10.13.20@KRBTEST.COM.keytab.
I20260925 02:33:05.912516 10292 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3SrHiO/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/postgres -l logfile start
2026-09-25 02:33:08.632 UTC [22132] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:33:08.632 UTC [22132] LOG: listening on IPv4 address "127.10.13.20", port 55187
2026-09-25 02:33:08.636 UTC [22132] LOG: listening on Unix socket "/tmp/.s.PGSQL.55187"
2026-09-25 02:33:08.670 UTC [22135] LOG: database system was shut down at 2026-09-25 02:33:06 UTC
2026-09-25 02:33:08.674 UTC [22132] LOG: database system is ready to accept connections
I20260925 02:33:09.627194 10292 mini_postgres.cc:96] Postgres bound to 55187
2026-09-25 02:33:09.634 UTC [22143] FATAL: database "slave" does not exist
127.10.13.20:55187 - accepting connections
I20260925 02:33:09.634640 10292 mini_ranger.cc:161] Starting Ranger...
I20260925 02:33:09.654251 10292 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:33:10.201575 10292 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:33:10.201645 10292 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-admin
2026-09-25 02:33:10,496 [I] DB FLAVOR :POSTGRES
2026-09-25 02:33:10,496 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:33:10,497 [I] Checking connection..
2026-09-25 02:33:10,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:33:10,756 [I] Checking connection passed.
2026-09-25 02:33:10,757 [I] --------- Verifying version history table ---------
2026-09-25 02:33:10,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:33:11,020 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:33:11,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:33:11,283 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:33:11,283 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:33:11,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:33:11.518 UTC [22207] WARNING: there is no transaction in progress
2026-09-25 02:33:11,525 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:33:11,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:33:11,817 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:33:11,817 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:33:11,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:33:12,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:33:12,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q8mg.c.gcp-upstream.internal','N') ;"
2026-09-25 02:33:12,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:33:12,774 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:33:12,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:33:13,021 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:33:13,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:33:13,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:33:13,520 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:33:13,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:33:14.543 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.561 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.580 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.596 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.614 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.633 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.701 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.709 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.725 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.735 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.746 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.762 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.773 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.780 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.788 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:14.795 UTC [22316] WARNING: there is no transaction in progress
2026-09-25 02:33:15,517 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:33:15,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:33:15,792 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:33:15,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:33:16,071 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:33:16,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:33:16,304 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:33:16,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:33:16,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:33:16,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q8mg.c.gcp-upstream.internal';"
2026-09-25 02:33:17,092 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:33:17,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:33:17,322 [I] DB_PATCHES have already been applied
I20260925 02:33:17.328199 10292 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-admin:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3SrHiO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:33:17.328265 10292 mini_ranger.cc:193] Using host: 127.10.13.20
I20260925 02:33:17.331866 10292 mini_ranger.cc:227] Ranger admin URL: http://127.10.13.20:44625
Sep 25, 2026 2:33:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:33:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:33:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:33:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:33:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:33:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:33:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:33:18.432415 10292 mini_ranger.cc:160] Time spent starting Ranger: real 8.798s user 0.000s sys 0.009s
Sep 25, 2026 2:33:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:33:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:33:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:33:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:33:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:33:39.398555 10292 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.13.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.13.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/rangerkms_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.13.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/rangerkms_127.10.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.13.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.13.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.13.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/HTTP_127.10.13.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.13.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/HTTP_127.10.13.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:33:39 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303619, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:33:39.491596 10292 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:33:39.501961 10292 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:33:40.502247 10292 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:33:40.502347 10292 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms
I20260925 02:33:40.502367 10292 mini_ranger_kms.cc:227] Using postgres at 127.10.13.20:55187
2026-09-25 02:33:40,669 [I] DB FLAVOR :POSTGRES
2026-09-25 02:33:40,669 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:33:40,669 [I] Checking connection
2026-09-25 02:33:40,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:33:40,928 [I] connection success
2026-09-25 02:33:40,928 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:33:40,928 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:33:40,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:33:41,212 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:33:41,213 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:33:41,213 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:33:41,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.13.20:55187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:33:41,507 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:33:41.977161 10292 mini_ranger_kms.cc:310] Created kms service
I20260925 02:33:42.204314 10292 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:33:42.248574 10292 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:33:42 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303619, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:42.630426 10292 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:33:42.630666 10292 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms:/tmp/dist-test-task3SrHiO/build/release/bin/postgresql.jar:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3SrHiO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:33:42.630697 10292 mini_ranger_kms.cc:240] Using host: 127.10.13.20
I20260925 02:33:42.634672 10292 mini_ranger_kms.cc:276] Ranger KMS PID: 22498
I20260925 02:33:42.634768 10292 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.13.20:60321
Sep 25, 2026 2:33:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:33:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:33:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:33:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:33:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:33:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:33:44.869411 10292 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.378s user 0.000s sys 0.150s
I20260925 02:33:44.869537 10292 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:33:44.869585 10292 mini_ranger_kms.cc:401] 127.10.13.20:60321/kms/v1/keys
Sep 25 02:33:44 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303624, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.13.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:33:45 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303624, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.13.20@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
Sep 25 02:33:45 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303624, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.13.20@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
Sep 25 02:33:48 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:33:48.185660 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45061
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:45061
--ranger_config_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:60321/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:48.292775 22549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:48.293111 22549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:48.293154 22549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:48.294394 22549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:48.294466 22549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:48.294490 22549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:48.294523 22549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:48.294549 22549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:48.296243 22549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.13.20:60321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:45061
--ranger_config_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:48.296552 22549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:48.296795 22549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:48.302452 22555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:48.302541 22554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:48.302433 22557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:48.302548 22549 server_base.cc:1034] running on GCE node
I20260925 02:33:48.303089 22549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:48.303432 22549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:48.304602 22549 hybrid_clock.cc:648] HybridClock initialized: now 1790303628304575 us; error 44 us; skew 500 ppm
Sep 25 02:33:48 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303628, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:48.311208 22549 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:33:48.311810 22549 webserver.cc:460] Webserver started at http://127.10.13.62:42721/ using document root <none> and password file <none>
I20260925 02:33:48.312080 22549 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:48.312124 22549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:48.312295 22549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:33:48 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303628, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:48.372761 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462"
format_stamp: "Formatted at 2026-09-25 02:33:48 on dist-test-slave-q8mg"
server_key: "a07b11bc2d5ccb7ff59599411d7438b4"
server_key_iv: "a9cb9da312ff572f513a57986c3bbb4b"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:48.373260 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462"
format_stamp: "Formatted at 2026-09-25 02:33:48 on dist-test-slave-q8mg"
server_key: "a07b11bc2d5ccb7ff59599411d7438b4"
server_key_iv: "a9cb9da312ff572f513a57986c3bbb4b"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:48.374768 22549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:48.390594 22564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:48.390867 22549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:48.390995 22549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462"
format_stamp: "Formatted at 2026-09-25 02:33:48 on dist-test-slave-q8mg"
server_key: "a07b11bc2d5ccb7ff59599411d7438b4"
server_key_iv: "a9cb9da312ff572f513a57986c3bbb4b"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:48.391100 22549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:48.405472 22549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:48.407809 22549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:48.407984 22549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:48.413406 22549 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:45061
I20260925 02:33:48.413498 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:45061 every 8 connection(s)
I20260925 02:33:48.413957 22549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:33:48.415022 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:48.417670 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Bootstrap starting.
I20260925 02:33:48.418356 22617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:48.418684 22617 log.cc:824] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:48.419463 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: No bootstrap required, opened a new log
I20260925 02:33:48.421823 22617 raft_consensus.cc:348] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } }
I20260925 02:33:48.422008 22617 raft_consensus.cc:374] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:48.422035 22617 raft_consensus.cc:729] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d921fdb4fbe2414fbbb49b9d2d9ba462, State: Initialized, Role: FOLLOWER
I20260925 02:33:48.422230 22617 consensus_queue.cc:260] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } }
I20260925 02:33:48.422312 22617 raft_consensus.cc:388] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:33:48.422360 22617 raft_consensus.cc:482] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:33:48.422415 22617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:48.423048 22617 raft_consensus.cc:504] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } }
I20260925 02:33:48.423190 22617 leader_election.cc:302] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d921fdb4fbe2414fbbb49b9d2d9ba462; no voters:
I20260925 02:33:48.423319 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 22549
I20260925 02:33:48.423420 22617 leader_election.cc:288] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:33:48.423471 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:33:48.423616 22621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:33:48.423727 10292 external_mini_cluster.cc:1428] Setting key 8a513b960776e155dfbfb36b375e129e
I20260925 02:33:48.423741 22617 sys_catalog.cc:564] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:33:48.423770 22621 raft_consensus.cc:686] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 1 LEADER]: Becoming Leader. State: Replica: d921fdb4fbe2414fbbb49b9d2d9ba462, State: Running, Role: LEADER
I20260925 02:33:48.423861 22621 consensus_queue.cc:237] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } }
I20260925 02:33:48.424185 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } } }
I20260925 02:33:48.424256 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:48.424384 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d921fdb4fbe2414fbbb49b9d2d9ba462. Latest consensus state: current_term: 1 leader_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } } }
I20260925 02:33:48.424424 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:48.425551 22617 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:33:48.429221 22626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:33:48.430272 22626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:33:48 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:48.437569 22626 catalog_manager.cc:1347] Generated new cluster ID: ac81f29f1df04641894ca984c859b4fa
I20260925 02:33:48.437634 22626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:33:48.449795 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:48.424485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.10.13.62:45061)
0925 02:33:48.424758 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:48.424769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:48.425096 (+ 327us) server_negotiation.cc:407] Connection header received
0925 02:33:48.425257 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:48.425266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:48.425336 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:48.425509 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:48.427847 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:48.430769 (+ 2922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:48.431985 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:48.432694 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:48.432785 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:48.437581 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:48.437588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:48.437588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:48.437607 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:48.447324 (+ 9717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:48.447849 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:48.447853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:48.447853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:48.447973 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:48.448200 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:48.448201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:48.448201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:48.448595 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:48.449290 (+ 695us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:48.449385 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 02:33:48.449525 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
I20260925 02:33:48.469285 22626 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:33:48.470235 22626 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:33:48.488270 22626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Generated new TSK 0
I20260925 02:33:48.488627 22626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:33:49.257820 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:60321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45061
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:49.400341 22662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:49.400614 22662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:49.400647 22662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:49.402436 22662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:49.402539 22662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:49.402591 22662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:33:49.404726 22662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.13.20:60321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:49.405321 22662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:49.405660 22662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:49.412315 22667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:49.413538 22668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:49.413827 22670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:49.414899 22662 server_base.cc:1034] running on GCE node
I20260925 02:33:49.415132 22662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:49.415433 22662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:49.416599 22662 hybrid_clock.cc:648] HybridClock initialized: now 1790303629416587 us; error 27 us; skew 500 ppm
Sep 25 02:33:49 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:49.420008 22662 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:33:49.420655 22662 webserver.cc:460] Webserver started at http://127.10.13.1:35513/ using document root <none> and password file <none>
I20260925 02:33:49.421034 22662 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:49.421155 22662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:49.421336 22662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:33:49 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:49.442243 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "a94dcd59a8234600a7780db48601b910"
format_stamp: "Formatted at 2026-09-25 02:33:49 on dist-test-slave-q8mg"
server_key: "f809e963cff1e4aca9b47c2e4bd654da"
server_key_iv: "180a1a24bb8139e3c5ea02f8b9488714"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:49.442657 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "a94dcd59a8234600a7780db48601b910"
format_stamp: "Formatted at 2026-09-25 02:33:49 on dist-test-slave-q8mg"
server_key: "f809e963cff1e4aca9b47c2e4bd654da"
server_key_iv: "180a1a24bb8139e3c5ea02f8b9488714"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:49.444284 22662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:33:49.467883 22679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:49.468206 22662 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260925 02:33:49.468330 22662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "a94dcd59a8234600a7780db48601b910"
format_stamp: "Formatted at 2026-09-25 02:33:49 on dist-test-slave-q8mg"
server_key: "f809e963cff1e4aca9b47c2e4bd654da"
server_key_iv: "180a1a24bb8139e3c5ea02f8b9488714"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:49.468453 22662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:49.504148 22662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:49.506772 22662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:49.506992 22662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:49.507309 22662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:49.507741 22662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:49.507829 22662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:49.507929 22662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:49.507999 22662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:49.515066 22662 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:34403
I20260925 02:33:49.515606 22662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:33:49.515763 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 22662
I20260925 02:33:49.515890 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:33:49.516122 10292 external_mini_cluster.cc:1428] Setting key d223c349e5dbce86839e560461fc7ef0
I20260925 02:33:49.525496 22792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:34403 every 8 connection(s)
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Sep 25 02:33:49 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:49.541932 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:49.526568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:59791 (local address 127.10.13.62:45061)
0925 02:33:49.529738 (+ 3170us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:49.529741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:49.529778 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:33:49.529809 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:49.529810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:49.529825 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:49.529898 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:49.530730 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:49.531497 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:49.534488 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:49.534825 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:49.534923 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:49.537901 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:49.537904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:49.537904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:49.537915 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:49.539991 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:49.540605 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:49.540606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:49.540607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:49.540674 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:49.541033 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:49.541034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:49.541035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:49.541192 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:49.541241 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:49.541697 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 02:33:49.541793 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3140,"thread_start_us":51,"threads_started":1}
I20260925 02:33:49.542531 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:49.526241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45061 (local address 127.10.13.1:59791)
0925 02:33:49.526453 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:49.526515 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:49.527216 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:49.530046 (+ 2830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:49.530051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:49.530305 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:49.530609 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:49.530616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:49.533007 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:49.533009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:49.534375 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:49.534380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:49.534946 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:49.534947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:49.535117 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:49.535757 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:49.535769 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:49.537760 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:49.540104 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:49.540113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:49.540117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:49.540501 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:49.540752 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:49.540753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:49.540753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:49.540941 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:49.541296 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:49.541303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:49.541526 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 02:33:49.542053 (+ 527us) client_negotiation.cc:240] Negotiation successful
0925 02:33:49.542237 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260925 02:33:49.543109 22793 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45061
I20260925 02:33:49.543216 22793 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:49.543424 22793 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:49.544102 22581 ts_manager.cc:194] Registered new tserver with Master: a94dcd59a8234600a7780db48601b910 (127.10.13.1:34403)
I20260925 02:33:49.545179 22581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:59791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:33:49.557401 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:60321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45061
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:33:49 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:33:49.703939 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:49.704324 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:49.704401 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:49.706228 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:49.706295 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:49.706334 22801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:33:49.708506 22801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.13.20:60321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:49.708774 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:49.709214 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:49.716674 22807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:49.716701 22806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:49.716835 22809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:49.718043 22801 server_base.cc:1034] running on GCE node
I20260925 02:33:49.718328 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:49.718642 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:49.719826 22801 hybrid_clock.cc:648] HybridClock initialized: now 1790303629719786 us; error 48 us; skew 500 ppm
Sep 25 02:33:49 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:49.723322 22801 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:33:49.723826 22801 webserver.cc:460] Webserver started at http://127.10.13.2:39255/ using document root <none> and password file <none>
I20260925 02:33:49.724109 22801 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:49.724164 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:49.724292 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:33:49 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:49.737222 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "46daf3cce4d749aea7ba9027591f3280"
format_stamp: "Formatted at 2026-09-25 02:33:49 on dist-test-slave-q8mg"
server_key: "a46cb0bef7d4e185e8b9a05a8f6a5f11"
server_key_iv: "3aa658125689bc36d1efbcc19b20f9c7"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:49.737649 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "46daf3cce4d749aea7ba9027591f3280"
format_stamp: "Formatted at 2026-09-25 02:33:49 on dist-test-slave-q8mg"
server_key: "a46cb0bef7d4e185e8b9a05a8f6a5f11"
server_key_iv: "3aa658125689bc36d1efbcc19b20f9c7"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:49.739774 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:33:49.753686 22816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:49.753904 22801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:49.753999 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "46daf3cce4d749aea7ba9027591f3280"
format_stamp: "Formatted at 2026-09-25 02:33:49 on dist-test-slave-q8mg"
server_key: "a46cb0bef7d4e185e8b9a05a8f6a5f11"
server_key_iv: "3aa658125689bc36d1efbcc19b20f9c7"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:49.754097 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:49.772684 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:49.775364 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:49.775579 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:49.775920 22801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:49.776393 22801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:49.776489 22801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:49.776572 22801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:49.776633 22801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:49.783936 22801 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:32861
I20260925 02:33:49.784512 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:33:49.787792 22930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:32861 every 8 connection(s)
I20260925 02:33:49.791358 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 22801
I20260925 02:33:49.791464 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:33:49.791692 10292 external_mini_cluster.cc:1428] Setting key 8e469a94ddfecbafc2938a70a540753b
Sep 25 02:33:49 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
I20260925 02:33:49.803591 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:49.788990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:52749 (local address 127.10.13.62:45061)
0925 02:33:49.789047 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:49.789049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:49.789198 (+ 149us) server_negotiation.cc:407] Connection header received
0925 02:33:49.789890 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:49.789892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:49.789906 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:49.789984 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:49.792939 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:49.793730 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:49.796058 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:49.796363 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:49.796425 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:49.799925 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:49.799927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:49.799927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:49.799935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:49.801705 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:49.802457 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:49.802458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:49.802459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:49.802519 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:49.802777 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:49.802778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:49.802778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:49.802908 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:49.802959 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:49.803381 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 02:33:49.803483 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:33:49.803720 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:49.788645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45061 (local address 127.10.13.2:52749)
0925 02:33:49.788988 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:49.789055 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:49.789755 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:49.790114 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:49.790119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:49.790364 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:49.790700 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:49.790714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:49.793880 (+ 3166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:49.793881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:49.795131 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:49.795133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:49.797338 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:49.797340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:49.797508 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:49.798142 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:49.798153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:49.799812 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:49.802052 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:49.802060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:49.802063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:49.802355 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:49.802584 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:49.802584 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:49.802585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:49.802709 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:49.802992 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:49.802998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:49.803182 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 02:33:49.803290 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:33:49.803478 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":113,"threads_started":1}
I20260925 02:33:49.804690 22931 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45061
I20260925 02:33:49.804805 22931 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:49.805051 22931 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:49.805614 22581 ts_manager.cc:194] Registered new tserver with Master: 46daf3cce4d749aea7ba9027591f3280 (127.10.13.2:32861)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
I20260925 02:33:49.806133 22581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:52749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:33:49.824712 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:60321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45061
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:49.989552 22936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:49.989939 22936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:49.990038 22936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:49.991771 22936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:49.991989 22936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:49.992079 22936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:33:49.994294 22936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.13.20:60321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:49.994755 22936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:49.995081 22936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:50.001794 22942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.001937 22941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.002295 22944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:50.003855 22936 server_base.cc:1034] running on GCE node
I20260925 02:33:50.004225 22936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:50.004539 22936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:50.011940 22936 hybrid_clock.cc:648] HybridClock initialized: now 1790303630011912 us; error 40 us; skew 500 ppm
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:50.015448 22936 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:33:50.016031 22936 webserver.cc:460] Webserver started at http://127.10.13.3:45455/ using document root <none> and password file <none>
I20260925 02:33:50.016321 22936 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:50.016382 22936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:50.016567 22936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:50.030769 22936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "fc68e75aea2b4886a40c2fbd300b60eb"
format_stamp: "Formatted at 2026-09-25 02:33:50 on dist-test-slave-q8mg"
server_key: "2419f93f7e930d84cab47917cf511a19"
server_key_iv: "b23f735bd8def204638e3020a908aa16"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:50.031337 22936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "fc68e75aea2b4886a40c2fbd300b60eb"
format_stamp: "Formatted at 2026-09-25 02:33:50 on dist-test-slave-q8mg"
server_key: "2419f93f7e930d84cab47917cf511a19"
server_key_iv: "b23f735bd8def204638e3020a908aa16"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:50.033203 22936 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:33:50.060812 22952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.061065 22936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:50.061169 22936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "fc68e75aea2b4886a40c2fbd300b60eb"
format_stamp: "Formatted at 2026-09-25 02:33:50 on dist-test-slave-q8mg"
server_key: "2419f93f7e930d84cab47917cf511a19"
server_key_iv: "b23f735bd8def204638e3020a908aa16"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:50.061268 22936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:50.111848 22936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:50.115082 22936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:50.115341 22936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:50.115679 22936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:50.116137 22936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:50.116380 22936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.116465 22936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:50.116526 22936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.125761 22936 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:33121
I20260925 02:33:50.126338 22936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:33:50.130123 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 22936
I20260925 02:33:50.130226 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:33:50.130420 10292 external_mini_cluster.cc:1428] Setting key 0e33d31554b927aee09e533de57b3033
I20260925 02:33:50.128863 23067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:33121 every 8 connection(s)
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:50.145594 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.129545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:44933 (local address 127.10.13.62:45061)
0925 02:33:50.129602 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:50.129604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:50.131862 (+ 2258us) server_negotiation.cc:407] Connection header received
0925 02:33:50.135033 (+ 3171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:50.135035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:50.135049 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:50.135111 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:50.135931 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.136715 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.138346 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.138641 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.138692 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.141916 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:50.141917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:50.141918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:50.141926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:50.143733 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.144280 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.144282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.144282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.144372 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.144720 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.144722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.144722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.144895 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:50.144963 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:50.145387 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 02:33:50.145492 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:33:50.146430 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.129536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45061 (local address 127.10.13.3:44933)
0925 02:33:50.131712 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:50.131767 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:50.134882 (+ 3115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:50.135210 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:50.135216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:50.135498 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:50.135801 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:50.135810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.136842 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.136843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:50.138221 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:50.138225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.138813 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.138814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:50.138963 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.139663 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:50.139675 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:50.141793 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:50.143834 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:50.143840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:50.143843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:50.144168 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:50.144458 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:50.144459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:50.144459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:50.144613 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:50.145009 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:50.145016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:50.145209 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 02:33:50.146000 (+ 791us) client_negotiation.cc:240] Negotiation successful
0925 02:33:50.146182 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2120,"thread_start_us":96,"threads_started":1}
I20260925 02:33:50.147140 23068 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45061
I20260925 02:33:50.147301 23068 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.147568 23068 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:50.148013 22581 ts_manager.cc:194] Registered new tserver with Master: fc68e75aea2b4886a40c2fbd300b60eb (127.10.13.3:33121)
I20260925 02:33:50.148488 22581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:44933
I20260925 02:33:50.153205 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:33:50.153949 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 22662
I20260925 02:33:50.162274 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 22801
I20260925 02:33:50.169486 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 22936
I20260925 02:33:50.177181 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 22549
I20260925 02:33:50.183949 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45061
--webserver_interface=127.10.13.62
--webserver_port=42721
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:45061
--ranger_config_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:60321/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.13.20:60321/kms with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:50.267007 23071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:50.267266 23071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:50.267303 23071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:50.268460 23071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:50.268536 23071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:50.268561 23071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:50.268585 23071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:50.268612 23071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:50.270257 23071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.13.20:60321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:45061
--ranger_config_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:50.270562 23071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:50.270800 23071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:50.276034 23077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.275990 23079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.276098 23076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:50.276139 23071 server_base.cc:1034] running on GCE node
I20260925 02:33:50.276564 23071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:50.276923 23071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:50.278087 23071 hybrid_clock.cc:648] HybridClock initialized: now 1790303630278058 us; error 41 us; skew 500 ppm
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:50.280759 23071 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:33:50.281303 23071 webserver.cc:460] Webserver started at http://127.10.13.62:42721/ using document root <none> and password file <none>
I20260925 02:33:50.281502 23071 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:50.281551 23071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:50.338361 23071 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:50.339205 23087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.339493 23071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:50.339623 23071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462"
format_stamp: "Formatted at 2026-09-25 02:33:48 on dist-test-slave-q8mg"
server_key: "a07b11bc2d5ccb7ff59599411d7438b4"
server_key_iv: "a9cb9da312ff572f513a57986c3bbb4b"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:50.340031 23071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:50.357686 23071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:50.359416 23071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:50.359611 23071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:50.364040 23071 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:45061
I20260925 02:33:50.364082 23139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:45061 every 8 connection(s)
I20260925 02:33:50.364495 23071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:33:50.365937 23140 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:33:50.366549 23140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Bootstrap starting.
I20260925 02:33:50.370289 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 23071
I20260925 02:33:50.370600 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:60321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:34403
--local_ip_for_outbound_sockets=127.10.13.1
--tserver_master_addrs=127.10.13.62:45061
--webserver_port=35513
--webserver_interface=127.10.13.1
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.13.20:60321/kms with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:33:50.375272 23140 log.cc:824] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:50.377818 23140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:33:50.378171 23140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Bootstrap complete.
I20260925 02:33:50.382515 23140 raft_consensus.cc:348] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } }
I20260925 02:33:50.382884 23140 raft_consensus.cc:729] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d921fdb4fbe2414fbbb49b9d2d9ba462, State: Initialized, Role: FOLLOWER
I20260925 02:33:50.383137 23140 consensus_queue.cc:260] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } }
I20260925 02:33:50.383212 23140 raft_consensus.cc:388] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:33:50.383258 23140 raft_consensus.cc:482] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:33:50.383299 23140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:33:50.384608 23140 raft_consensus.cc:504] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } }
I20260925 02:33:50.384718 23140 leader_election.cc:302] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d921fdb4fbe2414fbbb49b9d2d9ba462; no voters:
I20260925 02:33:50.384958 23140 leader_election.cc:288] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:33:50.384974 23144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:33:50.385155 23144 raft_consensus.cc:686] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [term 2 LEADER]: Becoming Leader. State: Replica: d921fdb4fbe2414fbbb49b9d2d9ba462, State: Running, Role: LEADER
I20260925 02:33:50.385254 23140 sys_catalog.cc:564] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:33:50.385308 23144 consensus_queue.cc:237] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } }
I20260925 02:33:50.385708 23146 sys_catalog.cc:455] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d921fdb4fbe2414fbbb49b9d2d9ba462. Latest consensus state: current_term: 2 leader_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } } }
I20260925 02:33:50.385815 23146 sys_catalog.cc:458] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:50.385710 23145 sys_catalog.cc:455] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d921fdb4fbe2414fbbb49b9d2d9ba462" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 45061 } } }
I20260925 02:33:50.385878 23145 sys_catalog.cc:458] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:50.386458 23151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:33:50.386809 23140 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:33:50.386978 23151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:33:50.387167 23151 catalog_manager.cc:1259] Loaded cluster ID: ac81f29f1df04641894ca984c859b4fa
I20260925 02:33:50.387212 23151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:33:50.388326 23151 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:33:50.388538 23151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Loaded TSK: 0
I20260925 02:33:50.388756 23151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:33:50.499805 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:50.500187 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:50.500268 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:50.502336 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:50.502468 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:50.502529 23142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:33:50.505039 23142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.13.20:60321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:34403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=35513
--tserver_master_addrs=127.10.13.62:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:50.505613 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:50.506003 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:50.513316 23177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.513538 23178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.513315 23180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:50.513370 23142 server_base.cc:1034] running on GCE node
I20260925 02:33:50.513927 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:50.514266 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:50.515456 23142 hybrid_clock.cc:648] HybridClock initialized: now 1790303630515433 us; error 42 us; skew 500 ppm
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:50.518556 23142 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:33:50.519250 23142 webserver.cc:460] Webserver started at http://127.10.13.1:35513/ using document root <none> and password file <none>
I20260925 02:33:50.519533 23142 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:50.519629 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:50.539266 23142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:33:50.540539 23188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.540776 23142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:50.540912 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "a94dcd59a8234600a7780db48601b910"
format_stamp: "Formatted at 2026-09-25 02:33:49 on dist-test-slave-q8mg"
server_key: "f809e963cff1e4aca9b47c2e4bd654da"
server_key_iv: "180a1a24bb8139e3c5ea02f8b9488714"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:50.541393 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:50.554962 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:50.557309 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:50.557489 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:50.557757 23142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:50.558146 23142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:50.558213 23142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.558281 23142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:50.558324 23142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.565559 23142 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:34403
I20260925 02:33:50.565961 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:33:50.566581 23301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:34403 every 8 connection(s)
I20260925 02:33:50.567459 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 23142
I20260925 02:33:50.567759 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:60321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:32861
--local_ip_for_outbound_sockets=127.10.13.2
--tserver_master_addrs=127.10.13.62:45061
--webserver_port=39255
--webserver_interface=127.10.13.2
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.13.20:60321/kms with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:50.582789 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.566903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:47595 (local address 127.10.13.62:45061)
0925 02:33:50.567236 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:50.567245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:50.567283 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:33:50.569952 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:50.569962 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:50.570039 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:50.570156 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:50.571072 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.572598 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.574359 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.575106 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.575195 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.578339 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:50.578345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:50.578345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:50.578361 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:50.580676 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.581237 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.581241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.581242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.581361 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.581623 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.581624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.581624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.581944 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:50.582017 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:50.582408 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 02:33:50.582541 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":92,"threads_started":1}
I20260925 02:33:50.583731 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.566754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45061 (local address 127.10.13.1:47595)
0925 02:33:50.567070 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:50.567114 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:50.569647 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:50.570394 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:50.570399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:50.570657 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:50.570939 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:50.570946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.572740 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.572741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:50.574245 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:50.574248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.575349 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.575350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:50.575503 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.576182 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:50.576194 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:50.578177 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:50.580782 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:50.580790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:50.580792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:50.581140 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:50.581434 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:50.581435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:50.581435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:50.581555 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:50.582053 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:50.582059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:50.582223 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 02:33:50.583281 (+ 1058us) client_negotiation.cc:240] Negotiation successful
0925 02:33:50.583507 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":126,"threads_started":1}
I20260925 02:33:50.585194 23302 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45061
I20260925 02:33:50.585294 23302 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.585510 23302 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
W20260925 02:33:50.586441 23302 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:45061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:33:50.586508 23302 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.586568 23302 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:50.586956 23302 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.587023 23302 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
W20260925 02:33:50.587322 23302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:33:50.674948 23306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:50.675220 23306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:50.675254 23306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:50.676985 23306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:50.677050 23306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:50.677093 23306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:33:50.679024 23306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.13.20:60321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:32861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=39255
--tserver_master_addrs=127.10.13.62:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:50.679286 23306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:50.679566 23306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:50.685664 23314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.685819 23311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.685868 23312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:50.686934 23306 server_base.cc:1034] running on GCE node
I20260925 02:33:50.687179 23306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:50.687455 23306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:50.688619 23306 hybrid_clock.cc:648] HybridClock initialized: now 1790303630688865 us; error 298 us; skew 500 ppm
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:50.691365 23306 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:33:50.691802 23306 webserver.cc:460] Webserver started at http://127.10.13.2:39255/ using document root <none> and password file <none>
I20260925 02:33:50.692055 23306 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:50.692138 23306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:50.719726 23306 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260925 02:33:50.720585 23322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.720800 23306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:33:50.720927 23306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "46daf3cce4d749aea7ba9027591f3280"
format_stamp: "Formatted at 2026-09-25 02:33:49 on dist-test-slave-q8mg"
server_key: "a46cb0bef7d4e185e8b9a05a8f6a5f11"
server_key_iv: "3aa658125689bc36d1efbcc19b20f9c7"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:50.721423 23306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:50.739291 23306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:50.741515 23306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:50.741703 23306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:50.741968 23306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:50.742344 23306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:50.742450 23306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.742496 23306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:50.742548 23306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.750512 23306 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:32861
I20260925 02:33:50.751075 23306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:33:50.752116 23435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:32861 every 8 connection(s)
I20260925 02:33:50.754288 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 23306
I20260925 02:33:50.754601 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.13.20:60321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:33121
--local_ip_for_outbound_sockets=127.10.13.3
--tserver_master_addrs=127.10.13.62:45061
--webserver_port=45455
--webserver_interface=127.10.13.3
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.13.20:60321/kms with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:50.769702 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.752999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:50641 (local address 127.10.13.62:45061)
0925 02:33:50.753048 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:50.753050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:50.757903 (+ 4853us) server_negotiation.cc:407] Connection header received
0925 02:33:50.758816 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:50.758818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:50.758834 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:50.758870 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:50.759859 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.760716 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.762489 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.762837 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.762909 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.766096 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:50.766098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:50.766098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:50.766108 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:50.767972 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.768485 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.768486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.768487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.768558 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.768865 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.768895 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.768895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.769047 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:50.769135 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:50.769504 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 02:33:50.769599 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:33:50.769836 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.752905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45061 (local address 127.10.13.2:50641)
0925 02:33:50.757725 (+ 4820us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:50.757789 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:50.758666 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:50.759111 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:50.759117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:50.759403 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:50.759729 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:50.759737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.760849 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.760850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:50.762370 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:50.762375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.763079 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.763080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:50.763320 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.764071 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:50.764085 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:50.765961 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:50.768070 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:50.768078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:50.768081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:50.768389 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:50.768634 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:50.768635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:50.768635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:50.768773 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:50.769126 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:50.769132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:50.769319 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 02:33:50.769427 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:33:50.769590 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4756,"thread_start_us":83,"threads_started":1}
I20260925 02:33:50.770315 23436 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45061
I20260925 02:33:50.770437 23436 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.770665 23436 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
W20260925 02:33:50.771471 23436 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:45061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:33:50.771556 23436 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.771626 23436 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:50.772002 23436 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.772075 23436 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
W20260925 02:33:50.772454 23436 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:33:50.862068 23439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:50.862417 23439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:50.862504 23439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:50.864341 23439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:50.864470 23439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:50.864542 23439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:33:50.866892 23439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.13.20:60321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:33121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=45455
--tserver_master_addrs=127.10.13.62:45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:50.867341 23439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:50.867655 23439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:50.874567 23447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.874591 23445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:50.874832 23444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:50.877106 23439 server_base.cc:1034] running on GCE node
I20260925 02:33:50.877382 23439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:50.877683 23439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:50.878867 23439 hybrid_clock.cc:648] HybridClock initialized: now 1790303630878832 us; error 46 us; skew 500 ppm
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:50.882261 23439 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:33:50.882848 23439 webserver.cc:460] Webserver started at http://127.10.13.3:45455/ using document root <none> and password file <none>
I20260925 02:33:50.883139 23439 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:50.883219 23439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:50.913403 23439 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260925 02:33:50.914345 23456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.914520 23439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:33:50.914608 23439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "fc68e75aea2b4886a40c2fbd300b60eb"
format_stamp: "Formatted at 2026-09-25 02:33:50 on dist-test-slave-q8mg"
server_key: "2419f93f7e930d84cab47917cf511a19"
server_key_iv: "b23f735bd8def204638e3020a908aa16"
server_key_version: "kuduclusterkey@0"
I20260925 02:33:50.915126 23439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:50.936259 23439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:50.938666 23439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:50.938840 23439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:50.939128 23439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:50.939522 23439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:50.939577 23439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.939620 23439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:50.939646 23439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:50.950052 23439 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:33121
I20260925 02:33:50.950193 23569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:33121 every 8 connection(s)
I20260925 02:33:50.950529 23439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:33:50.952208 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 23439
Sep 25 02:33:50 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:50.962358 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.951360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:40591 (local address 127.10.13.62:45061)
0925 02:33:50.951443 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:50.951446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:50.951710 (+ 264us) server_negotiation.cc:407] Connection header received
0925 02:33:50.952423 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:50.952424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:50.952440 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:50.952512 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:50.953340 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.954142 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.955677 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.955963 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.956028 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.958758 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:50.958759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:50.958759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:50.958767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:50.960657 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.961221 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.961223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.961223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.961285 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.961593 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.961594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.961594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.961724 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:50.961779 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:50.962180 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 02:33:50.962263 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:33:50.962553 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.952612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.10.13.62:45061)
0925 02:33:50.953661 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:50.953663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:50.953835 (+ 172us) server_negotiation.cc:407] Connection header received
0925 02:33:50.953905 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:50.953906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:50.953922 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:50.953996 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:50.955043 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.955526 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.956459 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.956723 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.956781 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.958359 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:50.958361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:50.958361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:50.958369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:50.960844 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.961295 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.961297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.961297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.961359 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:50.961564 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:50.961565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:50.961565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:50.961703 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:50.961884 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:50.962380 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 02:33:50.962461 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":69,"threads_started":1}
I20260925 02:33:50.963613 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:50.951346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:45061 (local address 127.10.13.3:40591)
0925 02:33:50.951558 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:50.951608 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:50.952284 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:50.952596 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:50.952601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:50.952912 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:50.953231 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:50.953238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.954245 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.954247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:50.955561 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:50.955565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:50.956072 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:50.956073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:50.956223 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:50.956856 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:50.956899 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:50.958636 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:50.960754 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:50.960762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:50.960765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:50.961122 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:50.961361 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:50.961362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:50.961362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:50.961514 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:50.961816 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:50.961823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:50.962014 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 02:33:50.962906 (+ 892us) client_negotiation.cc:240] Negotiation successful
0925 02:33:50.963260 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260925 02:33:50.964076 23570 heartbeater.cc:347] Connected to a master server at 127.10.13.62:45061
I20260925 02:33:50.964186 23570 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.964381 23570 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
W20260925 02:33:50.965147 23570 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:45061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:33:50.965240 23570 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.965324 23570 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:50.965881 23570 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:50.965998 23570 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
W20260925 02:33:50.966298 23570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 02:33:51 dist-test-slave-q8mg krb5kdc[22102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303631, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:51.587497 23302 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:51.587666 23302 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:51.588212 23104 ts_manager.cc:194] Registered new tserver with Master: a94dcd59a8234600a7780db48601b910 (127.10.13.1:34403)
I20260925 02:33:51.589314 23104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:47595
I20260925 02:33:51.772647 23436 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:51.772833 23436 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:51.773355 23104 ts_manager.cc:194] Registered new tserver with Master: 46daf3cce4d749aea7ba9027591f3280 (127.10.13.2:32861)
I20260925 02:33:51.773901 23104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:50641
Sep 25 02:33:51 dist-test-slave-q8mg krb5kdc[22102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303631, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for HTTP/127.10.13.20@KRBTEST.COM
I20260925 02:33:51.966519 23570 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:51.966675 23570 heartbeater.cc:511] Master 127.10.13.62:45061 requested a full tablet report, sending...
I20260925 02:33:51.967027 23104 ts_manager.cc:194] Registered new tserver with Master: fc68e75aea2b4886a40c2fbd300b60eb (127.10.13.3:33121)
I20260925 02:33:51.967487 23104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:40591
I20260925 02:33:51.976315 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:33:51.988039 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:51.977337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.10.13.62:45061)
0925 02:33:51.981154 (+ 3817us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:51.981155 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:51.981161 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:33:51.981181 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:51.981182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:51.981192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:51.981277 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:51.981837 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:51.982350 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:51.983106 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:51.983323 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:51.983414 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:51.984780 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:51.984782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:51.984782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:51.984791 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:51.986733 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:51.987142 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:51.987144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:51.987144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:51.987211 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:51.987413 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:51.987414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:51.987414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:51.987535 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:51.987639 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:51.987748 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:33:51.987834 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3797,"thread_start_us":74,"threads_started":1}
I20260925 02:33:51.990098 23104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:33:51.991751 23104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:33:51.998898 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:51.994990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.10.13.1:34403)
0925 02:33:51.995123 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:51.995126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:51.995135 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:33:51.995198 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:51.995200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:51.995284 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:51.995703 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:51.995799 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:51.996624 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:51.997945 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:51.998452 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:51.998583 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:51.998674 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:51.998771 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:33:51.998813 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260925 02:33:51.998904 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:51.994521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:34403 (local address 127.0.0.1:48944)
0925 02:33:51.994667 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:51.994707 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:51.994775 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:51.995544 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:51.995548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:51.995551 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:51.995623 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:51.995625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:51.996713 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:51.996714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:51.997843 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:51.997846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:51.998580 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:51.998581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:51.998663 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:51.998680 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:33:51.998757 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:33:51.998816 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260925 02:33:51.999943 23236 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab23cc94a2942aabf88a07d1d317821 (DEFAULT_TABLE table=test-table [id=26f24ed857a0483986799100d6f02dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:52.000320 23236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab23cc94a2942aabf88a07d1d317821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:52.005172 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:51.994774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:33121 (local address 127.0.0.1:41538)
0925 02:33:52.000942 (+ 6168us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:52.000954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:52.001019 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:52.001259 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:52.001260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:52.001263 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:52.001305 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:52.001306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.002922 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.002923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:52.003986 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:52.003989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.004922 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.004923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:52.004982 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.004986 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:33:52.005061 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:33:52.005101 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6148,"thread_start_us":72,"threads_started":1}
I20260925 02:33:52.005573 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:51.994892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.10.13.3:33121)
0925 02:33:51.995024 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:51.995029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:52.001091 (+ 6062us) server_negotiation.cc:407] Connection header received
0925 02:33:52.001139 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:52.001141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:52.001158 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:52.001239 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:52.001410 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.002826 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.004084 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.004716 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.004794 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.004864 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:52.005461 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 02:33:52.005503 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260925 02:33:52.007647 23487 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab23cc94a2942aabf88a07d1d317821 (DEFAULT_TABLE table=test-table [id=26f24ed857a0483986799100d6f02dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:52.008028 23487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab23cc94a2942aabf88a07d1d317821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:52.011564 23605 tablet_bootstrap.cc:492] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: Bootstrap starting.
I20260925 02:33:52.012265 23605 tablet_bootstrap.cc:654] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:52.012557 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:51.994538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:32861 (local address 127.0.0.1:55702)
0925 02:33:51.998978 (+ 4440us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:51.998989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:51.999035 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:52.004677 (+ 5642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:52.004678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:52.004680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:52.004716 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:52.004717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.006034 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.006035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:52.007126 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:52.007128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.012327 (+ 5199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.012328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:52.012400 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.012406 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:33:52.012436 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 02:33:52.012483 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4427,"thread_start_us":86,"threads_started":1}
I20260925 02:33:52.012640 23605 log.cc:824] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:52.013293 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:51.995143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.10.13.2:32861)
0925 02:33:52.004500 (+ 9357us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:52.004506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:52.004517 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:33:52.004552 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:52.004554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:52.004571 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:52.004781 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:52.004864 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.005933 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.011431 (+ 5498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.012221 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.013028 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.013137 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:52.013167 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 02:33:52.013215 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9318,"thread_start_us":45,"threads_started":1}
I20260925 02:33:52.013454 23605 tablet_bootstrap.cc:492] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: No bootstrap required, opened a new log
I20260925 02:33:52.013525 23605 ts_tablet_manager.cc:1397] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:33:52.013842 23604 tablet_bootstrap.cc:492] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910: Bootstrap starting.
I20260925 02:33:52.014092 23370 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab23cc94a2942aabf88a07d1d317821 (DEFAULT_TABLE table=test-table [id=26f24ed857a0483986799100d6f02dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:52.014457 23370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab23cc94a2942aabf88a07d1d317821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:52.014513 23604 tablet_bootstrap.cc:654] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:52.014801 23604 log.cc:824] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:52.015753 23605 raft_consensus.cc:348] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.015910 23605 raft_consensus.cc:374] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:52.015939 23605 raft_consensus.cc:729] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc68e75aea2b4886a40c2fbd300b60eb, State: Initialized, Role: FOLLOWER
I20260925 02:33:52.016049 23605 consensus_queue.cc:260] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.016299 23605 ts_tablet_manager.cc:1428] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:33:52.016356 23570 heartbeater.cc:503] Master 127.10.13.62:45061 was elected leader, sending a full tablet report...
I20260925 02:33:52.017771 23609 tablet_bootstrap.cc:492] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: Bootstrap starting.
I20260925 02:33:52.018364 23609 tablet_bootstrap.cc:654] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:52.018635 23609 log.cc:824] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:52.019421 23609 tablet_bootstrap.cc:492] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: No bootstrap required, opened a new log
I20260925 02:33:52.019503 23609 ts_tablet_manager.cc:1397] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:33:52.021548 23604 tablet_bootstrap.cc:492] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910: No bootstrap required, opened a new log
I20260925 02:33:52.021625 23604 ts_tablet_manager.cc:1397] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 02:33:52.021853 23609 raft_consensus.cc:348] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.022019 23609 raft_consensus.cc:374] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:52.022055 23609 raft_consensus.cc:729] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46daf3cce4d749aea7ba9027591f3280, State: Initialized, Role: FOLLOWER
I20260925 02:33:52.022223 23609 consensus_queue.cc:260] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.022538 23609 ts_tablet_manager.cc:1428] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:33:52.022646 23436 heartbeater.cc:503] Master 127.10.13.62:45061 was elected leader, sending a full tablet report...
I20260925 02:33:52.024255 23604 raft_consensus.cc:348] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.024379 23604 raft_consensus.cc:374] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:52.024399 23604 raft_consensus.cc:729] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94dcd59a8234600a7780db48601b910, State: Initialized, Role: FOLLOWER
I20260925 02:33:52.024524 23604 consensus_queue.cc:260] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.024796 23604 ts_tablet_manager.cc:1428] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:33:52.024806 23302 heartbeater.cc:503] Master 127.10.13.62:45061 was elected leader, sending a full tablet report...
I20260925 02:33:52.061177 23612 raft_consensus.cc:482] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:33:52.061342 23612 raft_consensus.cc:504] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.061754 23612 leader_election.cc:288] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46daf3cce4d749aea7ba9027591f3280 (127.10.13.2:32861), fc68e75aea2b4886a40c2fbd300b60eb (127.10.13.3:33121)
I20260925 02:33:52.066073 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:52.062228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:47237 (local address 127.10.13.2:32861)
0925 02:33:52.062276 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:52.062278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:52.062432 (+ 154us) server_negotiation.cc:407] Connection header received
0925 02:33:52.062545 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:52.062547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:52.062560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:52.062588 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:52.062814 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.063537 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.065078 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.065672 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.065719 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.065729 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:52.065956 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 02:33:52.065997 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:33:52.066082 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:52.062219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:41787 (local address 127.10.13.3:33121)
0925 02:33:52.062269 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:52.062271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:52.062454 (+ 183us) server_negotiation.cc:407] Connection header received
0925 02:33:52.062571 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:52.062572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:52.062587 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:52.062619 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:52.062852 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.063567 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.065093 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.065659 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.065709 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.065722 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:52.065959 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:33:52.066009 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:33:52.066428 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:52.061968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:32861 (local address 127.10.13.1:47237)
0925 02:33:52.062358 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:52.062369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:52.062472 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:52.062653 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:52.062653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:52.062655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:52.062747 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:52.062749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.063638 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.063639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:52.064974 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:52.064978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.065797 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.065798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:52.065869 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.065872 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:33:52.066290 (+ 418us) client_negotiation.cc:240] Negotiation successful
0925 02:33:52.066338 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":4,"thread_start_us":121,"threads_started":1}
I20260925 02:33:52.066744 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:52.062015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:33121 (local address 127.10.13.1:41787)
0925 02:33:52.062382 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:52.062393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:52.062486 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:52.062692 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:52.062692 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:52.062694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:52.062768 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:52.062770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.063677 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.063678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:52.064984 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:52.064987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.065784 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.065785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:52.065864 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.065868 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:33:52.066414 (+ 546us) client_negotiation.cc:240] Negotiation successful
0925 02:33:52.066465 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":88,"threads_started":1}
I20260925 02:33:52.066902 23383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab23cc94a2942aabf88a07d1d317821" candidate_uuid: "a94dcd59a8234600a7780db48601b910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46daf3cce4d749aea7ba9027591f3280" is_pre_election: true
I20260925 02:33:52.067081 23383 raft_consensus.cc:2454] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a94dcd59a8234600a7780db48601b910 in term 0.
I20260925 02:33:52.067219 23524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab23cc94a2942aabf88a07d1d317821" candidate_uuid: "a94dcd59a8234600a7780db48601b910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" is_pre_election: true
I20260925 02:33:52.067346 23190 leader_election.cc:302] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46daf3cce4d749aea7ba9027591f3280, a94dcd59a8234600a7780db48601b910; no voters:
I20260925 02:33:52.067392 23524 raft_consensus.cc:2454] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a94dcd59a8234600a7780db48601b910 in term 0.
W20260925 02:33:52.067448 23303 tablet.cc:2369] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:33:52.067480 23612 raft_consensus.cc:2781] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:33:52.067520 23612 raft_consensus.cc:482] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:33:52.067540 23612 raft_consensus.cc:3037] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:52.068384 23612 raft_consensus.cc:504] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.068662 23612 leader_election.cc:288] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [CANDIDATE]: Term 1 election: Requested vote from peers 46daf3cce4d749aea7ba9027591f3280 (127.10.13.2:32861), fc68e75aea2b4886a40c2fbd300b60eb (127.10.13.3:33121)
I20260925 02:33:52.068706 23383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab23cc94a2942aabf88a07d1d317821" candidate_uuid: "a94dcd59a8234600a7780db48601b910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46daf3cce4d749aea7ba9027591f3280"
I20260925 02:33:52.068804 23383 raft_consensus.cc:3037] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:52.068955 23524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab23cc94a2942aabf88a07d1d317821" candidate_uuid: "a94dcd59a8234600a7780db48601b910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc68e75aea2b4886a40c2fbd300b60eb"
I20260925 02:33:52.069029 23524 raft_consensus.cc:3037] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:52.069785 23383 raft_consensus.cc:2454] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a94dcd59a8234600a7780db48601b910 in term 1.
I20260925 02:33:52.069855 23524 raft_consensus.cc:2454] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a94dcd59a8234600a7780db48601b910 in term 1.
I20260925 02:33:52.069985 23190 leader_election.cc:302] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46daf3cce4d749aea7ba9027591f3280, a94dcd59a8234600a7780db48601b910; no voters:
I20260925 02:33:52.070086 23612 raft_consensus.cc:2781] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:33:52.070257 23612 raft_consensus.cc:686] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [term 1 LEADER]: Becoming Leader. State: Replica: a94dcd59a8234600a7780db48601b910, State: Running, Role: LEADER
I20260925 02:33:52.070366 23612 consensus_queue.cc:237] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } }
I20260925 02:33:52.071204 23104 catalog_manager.cc:5495] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 reported cstate change: term changed from 0 to 1, leader changed from <none> to a94dcd59a8234600a7780db48601b910 (127.10.13.1). New cstate: current_term: 1 leader_uuid: "a94dcd59a8234600a7780db48601b910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94dcd59a8234600a7780db48601b910" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 } health_report { overall_health: UNKNOWN } } }
I20260925 02:33:52.095324 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:52.092040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.10.13.1:34403)
0925 02:33:52.092084 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:52.092086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:52.092096 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:33:52.092164 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:52.092164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:52.092187 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:52.092265 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:33:52.092412 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.093151 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.094274 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:52.094577 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:52.094686 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:52.094879 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:33:52.095042 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:33:52.095140 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:52.095192 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:33:52.095223 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:33:52.097693 23524 raft_consensus.cc:1261] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 1 FOLLOWER]: Refusing update from remote peer a94dcd59a8234600a7780db48601b910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:33:52.098091 23612 consensus_queue.cc:1035] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc68e75aea2b4886a40c2fbd300b60eb" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 33121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:33:52.098268 23383 raft_consensus.cc:1261] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 1 FOLLOWER]: Refusing update from remote peer a94dcd59a8234600a7780db48601b910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:33:52.098537 23612 consensus_queue.cc:1035] T 0ab23cc94a2942aabf88a07d1d317821 P a94dcd59a8234600a7780db48601b910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46daf3cce4d749aea7ba9027591f3280" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:33:52.101974 23619 mvcc.cc:205] Tried to move back new op lower bound from 7333083677069275136 to 7333083676960948224. Current Snapshot: MvccSnapshot[applied={T|T < 7333083677069275136}]
I20260925 02:33:52.103811 23620 mvcc.cc:205] Tried to move back new op lower bound from 7333083677069275136 to 7333083676960948224. Current Snapshot: MvccSnapshot[applied={T|T < 7333083677069275136}]
I20260925 02:33:52.106462 23104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:33:52.106563 23104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:33:52.107295 23104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d921fdb4fbe2414fbbb49b9d2d9ba462: Sending DeleteTablet for 3 replicas of tablet 0ab23cc94a2942aabf88a07d1d317821
I20260925 02:33:52.107772 23487 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab23cc94a2942aabf88a07d1d317821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:52 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:41538
I20260925 02:33:52.107818 23236 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab23cc94a2942aabf88a07d1d317821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:52 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:48944
I20260925 02:33:52.108052 23370 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab23cc94a2942aabf88a07d1d317821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:52 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:55702
I20260925 02:33:52.108454 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 23142
I20260925 02:33:52.110100 23630 tablet_replica.cc:330] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: stopping tablet replica
I20260925 02:33:52.110219 23630 raft_consensus.cc:2229] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:33:52.110345 23630 raft_consensus.cc:2258] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:33:52.110545 23628 tablet_replica.cc:330] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: stopping tablet replica
I20260925 02:33:52.110644 23628 raft_consensus.cc:2229] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:33:52.110741 23628 raft_consensus.cc:2258] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:33:52.111217 23628 ts_tablet_manager.cc:1905] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:33:52.113299 23630 ts_tablet_manager.cc:1905] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:33:52.114936 23628 ts_tablet_manager.cc:1918] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:33:52.115022 23628 log.cc:1196] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/0ab23cc94a2942aabf88a07d1d317821
I20260925 02:33:52.115293 23628 ts_tablet_manager.cc:1939] T 0ab23cc94a2942aabf88a07d1d317821 P fc68e75aea2b4886a40c2fbd300b60eb: Deleting consensus metadata
I20260925 02:33:52.115744 23090 catalog_manager.cc:4841] TS fc68e75aea2b4886a40c2fbd300b60eb (127.10.13.3:33121): tablet 0ab23cc94a2942aabf88a07d1d317821 (table test-table [id=26f24ed857a0483986799100d6f02dc0]) successfully deleted
I20260925 02:33:52.115937 23630 ts_tablet_manager.cc:1918] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:33:52.116009 23630 log.cc:1196] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/0ab23cc94a2942aabf88a07d1d317821
I20260925 02:33:52.116268 23630 ts_tablet_manager.cc:1939] T 0ab23cc94a2942aabf88a07d1d317821 P 46daf3cce4d749aea7ba9027591f3280: Deleting consensus metadata
I20260925 02:33:52.116674 23089 catalog_manager.cc:4841] TS 46daf3cce4d749aea7ba9027591f3280 (127.10.13.2:32861): tablet 0ab23cc94a2942aabf88a07d1d317821 (table test-table [id=26f24ed857a0483986799100d6f02dc0]) successfully deleted
W20260925 02:33:52.118127 23088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:34403 (error 108)
I20260925 02:33:52.118685 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:52.118560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:34403 (local address 127.0.0.1:48974)
0925 02:33:52.118606 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:52.118644 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 02:33:52.118810 23088 catalog_manager.cc:4584] TS a94dcd59a8234600a7780db48601b910 (127.10.13.1:34403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ab23cc94a2942aabf88a07d1d317821: Network error: Client connection negotiation failed: client connection to 127.10.13.1:34403: connect: Connection refused (error 111)
I20260925 02:33:52.118893 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 23306
I20260925 02:33:52.125833 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 23439
I20260925 02:33:52.134195 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 23071
I20260925 02:33:52.140102 10292 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:33:52.485018 10292 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:33:52.485121 10292 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:33:52.830219 10292 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:33:52.830 UTC [22132] LOG: received smart shutdown request
2026-09-25 02:33:52.835 UTC [22132] LOG: background worker "logical replication launcher" (PID 22141) exited with exit code 1
2026-09-25 02:33:52.835 UTC [22136] LOG: shutting down
2026-09-25 02:33:53.029 UTC [22132] LOG: database system is shut down
2026-09-25T02:33:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48479 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:33:53 dist-test-slave-q8mg krb5kdc[23640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:33:53 dist-test-slave-q8mg krb5kdc[23640](info): set up 2 sockets
Sep 25 02:33:53 dist-test-slave-q8mg krb5kdc[23640](info): commencing operation
krb5kdc: starting...
W20260925 02:33:54.199816 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:33:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:33:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:33:54.328217 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:41381
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:41381,127.10.13.61:40663,127.10.13.60:41429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:54.411463 23653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:54.411682 23653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:54.411705 23653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:54.412884 23653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:54.412928 23653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:54.412949 23653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:54.412993 23653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:54.413023 23653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:54.414553 23653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:41381,127.10.13.61:40663,127.10.13.60:41429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:54.414827 23653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:54.415081 23653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:54.420073 23659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:54.420089 23661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:54.420112 23653 server_base.cc:1034] running on GCE node
W20260925 02:33:54.420061 23658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:54.420534 23653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:54.420853 23653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:54.422116 23653 hybrid_clock.cc:648] HybridClock initialized: now 1790303634422081 us; error 61 us; skew 500 ppm
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:54.430106 23653 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:33:54.430572 23653 webserver.cc:460] Webserver started at http://127.10.13.62:39997/ using document root <none> and password file <none>
I20260925 02:33:54.430799 23653 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:54.430846 23653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:54.430941 23653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:33:54.432003 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "1f1e03fb27284a8685e70bf4fce9b292"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "513b16f9271c2f06f525c08b1cb8b04c"
server_key_iv: "d6a8033f1245e7f8c37f24addaf99f77"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.432315 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "1f1e03fb27284a8685e70bf4fce9b292"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "513b16f9271c2f06f525c08b1cb8b04c"
server_key_iv: "d6a8033f1245e7f8c37f24addaf99f77"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.433907 23653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:33:54.434820 23668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:54.435103 23653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:33:54.435191 23653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "1f1e03fb27284a8685e70bf4fce9b292"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "513b16f9271c2f06f525c08b1cb8b04c"
server_key_iv: "d6a8033f1245e7f8c37f24addaf99f77"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.435302 23653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:54.457609 23653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:54.460222 23653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:54.460392 23653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:54.464828 23653 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:41381
I20260925 02:33:54.464921 23722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:41381 every 8 connection(s)
I20260925 02:33:54.465327 23653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:33:54.466275 23723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:54.467922 23723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:33:54.473713 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 23653
I20260925 02:33:54.473851 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:33:54.474049 10292 external_mini_cluster.cc:1428] Setting key 7b113cd30d36052cdf0feaa136929a66
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:54.478091 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.468692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41381 (local address 127.0.0.1:46176)
0925 02:33:54.468952 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.469009 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.469434 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.469754 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.469758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.470028 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.470195 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.470199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.471187 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.471189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.471914 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.471917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.472405 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.472406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.472471 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.473023 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.473032 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.474585 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.476532 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.476539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.476541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.476739 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.477034 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.477034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.477035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.477123 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.477485 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.477491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.477562 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.477674 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.477886 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":75,"thread_start_us":82,"threads_started":1}
I20260925 02:33:54.478092 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.468893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.10.13.62:41381)
0925 02:33:54.469087 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.469092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.469134 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:33:54.469591 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.469592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.469610 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.469689 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.470286 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.471091 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.472007 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.472324 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.472404 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.474677 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.474684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.474685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.474692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.476416 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.476822 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.476826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.476826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.476946 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.477188 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.477189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.477190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.477412 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.477514 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.477708 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.477828 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":129,"threads_started":1}
I20260925 02:33:54.478991 23723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:33:54.479378 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.479256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48056)
0925 02:33:54.479308 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.479354 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 02:33:54.479564 23669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
I20260925 02:33:54.480096 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.479972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48072)
0925 02:33:54.480040 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.480069 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:33:54.480266 23723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:40663: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.10.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab.
Entry for principal kudu/127.10.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab.
I20260925 02:33:54.499356 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.61:40663
--webserver_interface=127.10.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab
--principal=kudu/127.10.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:41381,127.10.13.61:40663,127.10.13.60:41429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:33:54.529536 23723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } attempt: 1
I20260925 02:33:54.530140 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.529922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48078)
0925 02:33:54.530044 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.530097 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 02:33:54.530561 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.530460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48082)
0925 02:33:54.530508 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.530538 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:33:54.530664 23723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:40663: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
W20260925 02:33:54.582818 23730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:54.583050 23730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:54.583073 23730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:54.584268 23730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:54.584313 23730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:54.584327 23730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:54.584340 23730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:54.584352 23730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:54.585873 23730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:41381,127.10.13.61:40663,127.10.13.60:41429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab
--principal=kudu/127.10.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.61:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:54.586090 23730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:54.586269 23730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:54.591293 23736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:54.591301 23735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:54.591503 23730 server_base.cc:1034] running on GCE node
W20260925 02:33:54.591558 23738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:54.591908 23730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:54.592211 23730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:54.593367 23730 hybrid_clock.cc:648] HybridClock initialized: now 1790303634593350 us; error 27 us; skew 500 ppm
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:54.596355 23730 init.cc:377] Logged in from keytab as kudu/127.10.13.61@KRBTEST.COM (short username kudu)
I20260925 02:33:54.596956 23730 webserver.cc:460] Webserver started at http://127.10.13.61:34663/ using document root <none> and password file <none>
I20260925 02:33:54.597204 23730 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:54.597251 23730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:54.597385 23730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:33:54.598505 23730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/data/instance:
uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "e765b75a9d1dbe73b86241b9a7d5391e"
server_key_iv: "515f602ee35eb811c146a8ebd8ec3565"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.598814 23723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } attempt: 2
I20260925 02:33:54.598878 23730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/wal/instance:
uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "e765b75a9d1dbe73b86241b9a7d5391e"
server_key_iv: "515f602ee35eb811c146a8ebd8ec3565"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.599309 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.599147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48094)
0925 02:33:54.599221 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.599263 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:33:54.599759 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.599641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48108)
0925 02:33:54.599702 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.599734 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 02:33:54.599880 23723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:40663: Network error: Client connection negotiation failed: client connection to 127.10.13.61:40663: connect: Connection refused (error 111)
I20260925 02:33:54.600618 23730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:33:54.601517 23745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:54.601732 23730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:33:54.601830 23730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/wal
uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "e765b75a9d1dbe73b86241b9a7d5391e"
server_key_iv: "515f602ee35eb811c146a8ebd8ec3565"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.601904 23730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:54.615316 23730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:54.617849 23730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:54.618033 23730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:54.622694 23730 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.61:40663
I20260925 02:33:54.622743 23797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.61:40663 every 8 connection(s)
I20260925 02:33:54.623140 23730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
I20260925 02:33:54.624065 23798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:54.624737 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 23730
I20260925 02:33:54.624948 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-1/wal/instance
I20260925 02:33:54.625183 10292 external_mini_cluster.cc:1428] Setting key cd4f9d70b737945992486b938dff1334
I20260925 02:33:54.625844 23798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.13.60@KRBTEST.COM; defaulting to no policy
I20260925 02:33:54.636032 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.626604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.10.13.62:41381)
0925 02:33:54.626655 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.626657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.627024 (+ 367us) server_negotiation.cc:407] Connection header received
0925 02:33:54.627798 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.627800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.627816 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.627878 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.628851 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.629410 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.630453 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.630680 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.630762 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.633090 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.633092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.633092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.633099 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.634584 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.635018 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.635019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.635020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.635080 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.635339 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.635340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.635341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.635463 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.635560 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.635843 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.635933 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:33:54.636245 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.626606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41381 (local address 127.0.0.1:46182)
0925 02:33:54.626828 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.626889 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.627673 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.627969 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.627973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.628414 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.628728 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.628735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.629549 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.629550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.630352 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.630354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.630772 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.630774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.630904 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.631520 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.631529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.632967 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.634673 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.634681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.634684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.634915 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.635157 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.635159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.635159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.635268 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.635552 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.635560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.635707 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.635822 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.635995 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.60@KRBTEST.COM" created.
I20260925 02:33:54.637128 23798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:33:54.644773 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.637410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48116)
0925 02:33:54.637469 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.637481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.637527 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.637802 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.637803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.637973 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.638035 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.638037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.639127 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.639128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.639716 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.639718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.640189 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.640190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.640259 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.640711 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.640716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.641882 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.643713 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.643714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.643714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.643873 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.644092 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.644093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.644093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.644147 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.644443 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.644444 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.644490 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.644555 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.644655 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 02:33:54.644836 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.637481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.10.13.61:40663)
0925 02:33:54.637648 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.637653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.637681 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:33:54.637699 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.637701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.637712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.637802 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.638135 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.639023 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.639803 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.640111 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.640192 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.641988 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.641997 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.641997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.642003 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.643610 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.643963 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.643966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.643967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.644026 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.644228 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.644229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.644229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.644372 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.644467 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.644584 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.644708 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260925 02:33:54.645533 23798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:33:54.645943 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.645816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48088)
0925 02:33:54.645867 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.645913 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:33:54.646086 23749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
I20260925 02:33:54.646517 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.646409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48104)
0925 02:33:54.646460 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.646492 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 02:33:54.646685 23798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:41429: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab.
Entry for principal kudu/127.10.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab.
I20260925 02:33:54.653482 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.60:41429
--webserver_interface=127.10.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab
--principal=kudu/127.10.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:41381,127.10.13.61:40663,127.10.13.60:41429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:33:54.691074 23723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } attempt: 3
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:33:54.696040 23798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } attempt: 1
I20260925 02:33:54.696436 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.696308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48106)
0925 02:33:54.696371 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.696408 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 02:33:54.697098 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.696974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48108)
0925 02:33:54.697040 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.697075 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:33:54.697286 23798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:41429: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
I20260925 02:33:54.698467 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.691522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48118)
0925 02:33:54.691590 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.691606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.691676 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.691852 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.691853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.692024 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.692091 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.692093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.692772 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.692774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.693342 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.693345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.693744 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.693745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.693850 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.694357 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.694362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.695562 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.697517 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.697518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.697518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.697717 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.697926 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.697926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.697926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.697974 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.698231 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.698232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.698274 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.698326 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.698404 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 02:33:54.698534 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.691625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.10.13.61:40663)
0925 02:33:54.691697 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.691699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.691722 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:33:54.691771 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.691772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.691783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.691844 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.692176 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.692685 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.693440 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.693666 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.693746 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.695651 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.695652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.695653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.695662 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.697422 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.697798 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.697800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.697800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.697862 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.698032 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.698033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.698033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.698166 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.698234 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.698374 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.698453 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:33:54.698904 23723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:33:54.699240 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.699154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48116)
0925 02:33:54.699189 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.699218 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 02:33:54.699577 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.699497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48120)
0925 02:33:54.699539 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.699562 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:33:54.699681 23723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:41429: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
I20260925 02:33:54.730854 23723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } attempt: 1
I20260925 02:33:54.731400 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.731254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48122)
0925 02:33:54.731328 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.731370 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:33:54.731801 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.731719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48126)
0925 02:33:54.731761 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.731786 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 02:33:54.731930 23723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:41429: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
W20260925 02:33:54.739620 23805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:54.739811 23805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:54.739840 23805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:54.740964 23805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:54.741004 23805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:54.741019 23805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:54.741031 23805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:54.741043 23805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:54.742465 23805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:41381,127.10.13.61:40663,127.10.13.60:41429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab
--principal=kudu/127.10.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.60:41429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:54.742657 23805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:54.742837 23805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:54.747743 23813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:54.747767 23810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:54.747743 23811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:54.748008 23805 server_base.cc:1034] running on GCE node
I20260925 02:33:54.748279 23805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:54.748594 23805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:54.749742 23805 hybrid_clock.cc:648] HybridClock initialized: now 1790303634749731 us; error 31 us; skew 500 ppm
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:54.752264 23805 init.cc:377] Logged in from keytab as kudu/127.10.13.60@KRBTEST.COM (short username kudu)
I20260925 02:33:54.752731 23805 webserver.cc:460] Webserver started at http://127.10.13.60:32861/ using document root <none> and password file <none>
I20260925 02:33:54.753039 23805 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:54.753110 23805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:54.753270 23805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:33:54.754359 23805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/data/instance:
uuid: "65bfc0292e4f4ddd9202dd8641eca87d"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "4675841f75467a01072ec1d3d5ab9494"
server_key_iv: "2c4bd1c46675acf686416e73ea7813d9"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.754743 23805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/wal/instance:
uuid: "65bfc0292e4f4ddd9202dd8641eca87d"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "4675841f75467a01072ec1d3d5ab9494"
server_key_iv: "2c4bd1c46675acf686416e73ea7813d9"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.756150 23805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:33:54.756987 23820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:54.757165 23805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:54.757272 23805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/wal
uuid: "65bfc0292e4f4ddd9202dd8641eca87d"
format_stamp: "Formatted at 2026-09-25 02:33:54 on dist-test-slave-q8mg"
server_key: "4675841f75467a01072ec1d3d5ab9494"
server_key_iv: "2c4bd1c46675acf686416e73ea7813d9"
server_key_version: "encryptionkey@0"
I20260925 02:33:54.757361 23805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:54.765461 23798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } attempt: 2
I20260925 02:33:54.766135 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.765963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48136)
0925 02:33:54.766050 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.766102 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:33:54.766542 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.766429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48146)
0925 02:33:54.766482 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.766518 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:33:54.766630 23798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:41429: Network error: Client connection negotiation failed: client connection to 127.10.13.60:41429: connect: Connection refused (error 111)
I20260925 02:33:54.777988 23805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:54.780579 23805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:54.780755 23805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:54.784721 23805 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.60:41429
I20260925 02:33:54.784794 23872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.60:41429 every 8 connection(s)
I20260925 02:33:54.785207 23805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
I20260925 02:33:54.786147 23873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:54.787734 23873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:33:54.789197 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 23805
I20260925 02:33:54.789324 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790303478591324-10292-0/minicluster-data/master-2/wal/instance
I20260925 02:33:54.789525 10292 external_mini_cluster.cc:1428] Setting key 6c5fae355f6c502b2d04ebf9ff81bebe
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
I20260925 02:33:54.797829 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.788523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.10.13.62:41381)
0925 02:33:54.788622 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.788624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.788773 (+ 149us) server_negotiation.cc:407] Connection header received
0925 02:33:54.789337 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.789339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.789351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.789399 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.790241 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.790799 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.791842 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.792033 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.792084 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.794600 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.794601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.794601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.794608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.796171 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.796563 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.796564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.796565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.796617 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.797030 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.797032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.797032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.797163 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.797226 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.797606 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.797699 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:33:54.797834 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.788410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:41381 (local address 127.0.0.1:46186)
0925 02:33:54.788639 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.788687 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.789227 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.789477 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.789481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.789777 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.790143 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.790149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.790894 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.790895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.791725 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.791727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.792114 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.792115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.792265 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.792857 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.792909 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.794498 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.796258 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.796264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.796266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.796496 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.796699 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.796700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.796701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.796854 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.797246 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.797251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.797419 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.797528 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.797641 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260925 02:33:54.798448 23873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:33:54.798765 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.789990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.10.13.60:41429)
0925 02:33:54.790171 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.790181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.790216 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:33:54.790240 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.790242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.790268 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.790351 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.790928 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.791703 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.792639 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.793097 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.793152 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.795411 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.795426 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.795426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.795432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.797392 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.797859 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.797862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.797862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.797919 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.798166 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.798167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.798167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.798339 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.798500 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.798572 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.798648 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":140,"thread_start_us":114,"threads_started":1}
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:54.806632 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.798829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.10.13.61:40663)
0925 02:33:54.798959 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.798961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.798969 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:33:54.798992 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.798993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.799022 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.799091 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.799404 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.800265 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.801293 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.801610 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.801692 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.803756 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.803757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.803757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.803765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.805355 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.805755 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.805756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.805756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.805855 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.806088 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.806089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.806089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.806202 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.806253 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.806485 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.806558 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 02:33:54.806632 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.798765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:40663 (local address 127.0.0.1:48126)
0925 02:33:54.798858 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.798868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.798917 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.799101 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.799102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.799215 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.799290 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.799292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.800353 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.800354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.801178 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.801181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.801695 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.801696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.801753 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.802227 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.802231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.803663 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.805426 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.805427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.805428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.805675 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.805924 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.805925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.805925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.806016 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.806273 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.806273 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.806322 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.806405 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.806506 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 02:33:54.807077 23873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:33:54.807083 23723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } attempt: 2
I20260925 02:33:54.808095 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.799666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.10.13.62:41381)
0925 02:33:54.799722 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.799723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.799731 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:33:54.799753 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.799753 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.799762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.799814 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.800439 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.801164 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.802044 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.802236 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.802313 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.804847 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.804848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.804849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.804857 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.806631 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.807196 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.807197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.807197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.807259 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.807480 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.807482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.807482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.807615 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.807835 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.807873 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.807990 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:33:54.816224 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.807361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48172)
0925 02:33:54.807407 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.807417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.807461 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.807806 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.807806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.807959 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.807999 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.808001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.808981 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.808982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.809493 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.809495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.809970 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.809971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.810025 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.810653 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.810658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.812063 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.815084 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.815084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.815085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.815290 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.815543 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.815544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.815544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.815617 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.815899 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.815899 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.815939 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.816027 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.816124 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:33:54.816462 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.807341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48160)
0925 02:33:54.807375 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.807387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.807420 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.807797 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.807798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.808015 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.808060 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.808062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.809026 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.809027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.809913 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.809916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.810426 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.810428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.810490 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.810857 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.810862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.812702 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.815058 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.815059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.815060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.815554 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.815760 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.815761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.815761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.815820 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.816137 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.816139 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.816185 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.816257 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.816365 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 02:33:54.816988 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.807486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.10.13.60:41429)
0925 02:33:54.807678 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.807680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.807686 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:33:54.807699 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.807700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.807707 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.807768 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.808071 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.808900 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.809562 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.809879 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.809950 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.812171 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.812172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.812173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.812179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.814997 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.815367 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.815368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.815369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.815462 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.815677 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.815678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.815679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.815827 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.815907 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.816050 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.816826 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":732,"server-negotiator.queue_time_us":171,"thread_start_us":137,"threads_started":1}
I20260925 02:33:54.817345 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.807465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.10.13.60:41429)
0925 02:33:54.807660 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.807663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.807671 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:33:54.807693 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.807694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.807704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.807937 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.808133 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.808956 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.810001 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.810297 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.810391 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.812815 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.812816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.812817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.812824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.814988 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.815623 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.815624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.815624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.815671 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.815898 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.815899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.815900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.816041 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.816134 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.816967 (+ 833us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.817246 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":175}
I20260925 02:33:54.817582 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.808733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.10.13.61:40663)
0925 02:33:54.808767 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.808770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.808777 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:33:54.808836 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.808837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.808846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.808930 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.809525 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.810043 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.811328 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.811640 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.811714 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.814488 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.814489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.814490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.814498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.816188 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.816557 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.816559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.816559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.816616 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.816817 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.816818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.816818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.816974 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.817377 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.817418 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.817502 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:33:54.818755 23873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d: Bootstrap starting.
I20260925 02:33:54.819399 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:33:54.819583 23873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:54.819798 23723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292: Bootstrap starting.
I20260925 02:33:54.819912 23873 log.cc:824] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:54.820528 23723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:54.820837 23873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d: No bootstrap required, opened a new log
I20260925 02:33:54.820873 23723 log.cc:824] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:54.821789 23723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292: No bootstrap required, opened a new log
I20260925 02:33:54.823123 23873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.823287 23873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:54.823328 23873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65bfc0292e4f4ddd9202dd8641eca87d, State: Initialized, Role: FOLLOWER
I20260925 02:33:54.823472 23873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.823769 23873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:33:54.823879 23723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.824007 23723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:54.824047 23723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1e03fb27284a8685e70bf4fce9b292, State: Initialized, Role: FOLLOWER
I20260925 02:33:54.824157 23723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.824473 23723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:33:54.826009 23902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:33:54.826100 23902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 02:33:54.826930 23911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:33:54.826989 23911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:33:54.827155 23891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.827232 23891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:33:54.826004 23805 master_runner.cc:186] Error getting master registration for 127.10.13.62:41381: OK, instance_id { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" instance_seqno: 1790303634463541 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:33:54.828483 23805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:33:54.827283 23653 master_runner.cc:186] Error getting master registration for 127.10.13.61:40663: OK, instance_id { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" instance_seqno: 1790303634621349 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:33:54.829823 23653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:33:54.829660 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.830082 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:33:54.831204 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.821019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.10.13.60:41429)
0925 02:33:54.821185 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.821187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.821289 (+ 102us) server_negotiation.cc:407] Connection header received
0925 02:33:54.821390 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.821391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.821403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.821433 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.822286 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.822976 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.824451 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.825147 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.825201 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.827148 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.827150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.827150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.827158 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.829094 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.829994 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.829995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.829995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.830358 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.830586 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.830587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.830587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.830718 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.830794 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.831034 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.831121 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260925 02:33:54.832396 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.820969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.10.13.62:41381)
0925 02:33:54.821062 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.821064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.821310 (+ 246us) server_negotiation.cc:407] Connection header received
0925 02:33:54.821422 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.821423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.821442 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.821471 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.822204 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.822926 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.824038 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.824366 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.824414 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.829252 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.829254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.829255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.829262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.830847 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.831279 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.831280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.831280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.831338 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.831543 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.831544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.831544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.831674 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.831772 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.832239 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.832314 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:33:54.840718 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.833214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.10.13.61:40663)
0925 02:33:54.833264 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.833266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.833291 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:33:54.833362 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.833362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.833373 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.833443 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.834093 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.834828 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.835816 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.836127 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.836205 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.837652 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.837653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.837654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.837661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.839470 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.839848 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.839849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.839849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.839915 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.840147 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.840149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.840149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.840285 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.840400 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.840537 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.840618 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:33:54.857776 23798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } attempt: 3
Sep 25 02:33:54 dist-test-slave-q8mg krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303634, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
I20260925 02:33:54.866669 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.858101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:41429 (local address 127.0.0.1:48184)
0925 02:33:54.858152 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:54.858164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:54.858201 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:54.858357 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:54.858358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:54.858547 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:54.858604 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.858606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.859586 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.859587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.860444 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:54.860447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.860972 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.860973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:54.861035 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.861551 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:54.861557 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:54.863115 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:54.865134 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.865135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.865136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.865367 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.865961 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:54.865962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:54.865962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:54.866038 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:54.866330 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:54.866331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:54.866383 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 02:33:54.866459 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:33:54.866561 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:33:54.866677 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:54.858175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.10.13.60:41429)
0925 02:33:54.858217 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:54.858219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:54.858228 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:33:54.858254 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:54.858254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:54.858268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:54.858341 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:54.858691 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.859478 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.860527 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:54.860895 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:54.860973 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:54.863211 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:54.863213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:54.863213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:54.863221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:54.865047 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.865825 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.865827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.865828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.865893 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:54.866113 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:54.866114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:54.866114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:54.866254 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:54.866323 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:54.866498 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:33:54.866581 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:33:54.868909 23798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1: Bootstrap starting.
I20260925 02:33:54.869696 23798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:54.870038 23798 log.cc:824] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:54.870949 23798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1: No bootstrap required, opened a new log
I20260925 02:33:54.873374 23798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.873515 23798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:54.873551 23798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6a4cae1e6b49619d3b1626dd9e1cb1, State: Initialized, Role: FOLLOWER
I20260925 02:33:54.873672 23798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.873972 23914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.874104 23914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:33:54.874027 23798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:33:54.876046 23925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:33:54.876156 23925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:33:54.878229 23730 master_runner.cc:263] No leader master found from master 2d6a4cae1e6b49619d3b1626dd9e1cb1
I20260925 02:33:54.878275 23730 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:33:54.930895 23891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:33:54.931046 23891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.931313 23891 leader_election.cc:288] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d6a4cae1e6b49619d3b1626dd9e1cb1 (127.10.13.61:40663), 65bfc0292e4f4ddd9202dd8641eca87d (127.10.13.60:41429)
I20260925 02:33:54.931712 23847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f1e03fb27284a8685e70bf4fce9b292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" is_pre_election: true
I20260925 02:33:54.931890 23847 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1e03fb27284a8685e70bf4fce9b292 in term 0.
I20260925 02:33:54.932381 23772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f1e03fb27284a8685e70bf4fce9b292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" is_pre_election: true
I20260925 02:33:54.932533 23772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1e03fb27284a8685e70bf4fce9b292 in term 0.
I20260925 02:33:54.932741 23672 leader_election.cc:302] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1e03fb27284a8685e70bf4fce9b292, 65bfc0292e4f4ddd9202dd8641eca87d; no voters:
I20260925 02:33:54.932961 23891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:33:54.933022 23891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:33:54.933048 23891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:54.934018 23891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.934154 23891 leader_election.cc:288] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [CANDIDATE]: Term 1 election: Requested vote from peers 2d6a4cae1e6b49619d3b1626dd9e1cb1 (127.10.13.61:40663), 65bfc0292e4f4ddd9202dd8641eca87d (127.10.13.60:41429)
I20260925 02:33:54.934585 23772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f1e03fb27284a8685e70bf4fce9b292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1"
I20260925 02:33:54.934688 23772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:54.935604 23772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1e03fb27284a8685e70bf4fce9b292 in term 1.
I20260925 02:33:54.935904 23669 leader_election.cc:302] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1e03fb27284a8685e70bf4fce9b292, 2d6a4cae1e6b49619d3b1626dd9e1cb1; no voters:
I20260925 02:33:54.936259 23847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f1e03fb27284a8685e70bf4fce9b292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65bfc0292e4f4ddd9202dd8641eca87d"
I20260925 02:33:54.936362 23847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:54.938549 23847 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1e03fb27284a8685e70bf4fce9b292 in term 1.
I20260925 02:33:54.939440 23891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:33:54.939563 23891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [term 1 LEADER]: Becoming Leader. State: Replica: 1f1e03fb27284a8685e70bf4fce9b292, State: Running, Role: LEADER
I20260925 02:33:54.939646 23891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } }
I20260925 02:33:54.940052 23891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f1e03fb27284a8685e70bf4fce9b292. Latest consensus state: current_term: 1 leader_uuid: "1f1e03fb27284a8685e70bf4fce9b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.940143 23891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:54.940583 23929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:33:54.940829 23929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:33:54.943085 23847 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [term 1 FOLLOWER]: Refusing update from remote peer 1f1e03fb27284a8685e70bf4fce9b292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:33:54.943485 23927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:33:54.944913 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f1e03fb27284a8685e70bf4fce9b292. Latest consensus state: current_term: 1 leader_uuid: "1f1e03fb27284a8685e70bf4fce9b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.945012 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:33:54.945472 23772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [term 1 FOLLOWER]: Refusing update from remote peer 1f1e03fb27284a8685e70bf4fce9b292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:33:54.945827 23927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:33:54.947148 23914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f1e03fb27284a8685e70bf4fce9b292. Latest consensus state: current_term: 1 leader_uuid: "1f1e03fb27284a8685e70bf4fce9b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.947238 23914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:33:54.948309 23927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f1e03fb27284a8685e70bf4fce9b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.948393 23927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:54.948863 23914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f1e03fb27284a8685e70bf4fce9b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.948967 23914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:33:54.949237 23930 mvcc.cc:205] Tried to move back new op lower bound from 7333083688724508672 to 7333083688713142272. Current Snapshot: MvccSnapshot[applied={T|T < 7333083688724508672}]
I20260925 02:33:54.950058 23933 mvcc.cc:205] Tried to move back new op lower bound from 7333083688724508672 to 7333083688713142272. Current Snapshot: MvccSnapshot[applied={T|T < 7333083688724508672}]
I20260925 02:33:54.950896 23891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f1e03fb27284a8685e70bf4fce9b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.950989 23891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:54.951395 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f1e03fb27284a8685e70bf4fce9b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1e03fb27284a8685e70bf4fce9b292" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41381 } } peers { permanent_uuid: "2d6a4cae1e6b49619d3b1626dd9e1cb1" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 40663 } } peers { permanent_uuid: "65bfc0292e4f4ddd9202dd8641eca87d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 41429 } } }
I20260925 02:33:54.951483 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:33:54.952485 23931 mvcc.cc:205] Tried to move back new op lower bound from 7333083688724508672 to 7333083688713142272. Current Snapshot: MvccSnapshot[applied={T|T < 7333083688724508672}]
I20260925 02:33:54.952726 23929 catalog_manager.cc:1347] Generated new cluster ID: 32d443b346b54e1e884b84ed3f5c2871
I20260925 02:33:54.952773 23929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:33:54.976832 23929 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:33:54.977737 23929 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:33:54.990314 23929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f1e03fb27284a8685e70bf4fce9b292: Generated new TSK 0
I20260925 02:33:54.990621 23929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:33:55.826624 23902 catalog_manager.cc:1259] Loaded cluster ID: 32d443b346b54e1e884b84ed3f5c2871
I20260925 02:33:55.826709 23902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d: loading cluster ID for follower catalog manager: success
I20260925 02:33:55.827668 23902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d: acquiring CA information for follower catalog manager: success
I20260925 02:33:55.828020 23902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65bfc0292e4f4ddd9202dd8641eca87d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:33:55.876675 23925 catalog_manager.cc:1259] Loaded cluster ID: 32d443b346b54e1e884b84ed3f5c2871
I20260925 02:33:55.876755 23925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1: loading cluster ID for follower catalog manager: success
I20260925 02:33:55.877697 23925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1: acquiring CA information for follower catalog manager: success
I20260925 02:33:55.878036 23925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d6a4cae1e6b49619d3b1626dd9e1cb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:33:56.049125 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 23653
I20260925 02:33:56.056025 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 23730
I20260925 02:33:56.061724 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 23805
2026-09-25T02:33:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2907 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:33:56 dist-test-slave-q8mg krb5kdc[23943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:33:56 dist-test-slave-q8mg krb5kdc[23943](info): set up 2 sockets
Sep 25 02:33:56 dist-test-slave-q8mg krb5kdc[23943](info): commencing operation
krb5kdc: starting...
W20260925 02:33:57.179467 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:33:57 dist-test-slave-q8mg krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:33:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:33:57.303368 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:43793
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:43793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:57.390355 23956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:57.390594 23956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:57.390618 23956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:57.391788 23956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:57.391824 23956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:57.391839 23956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:57.391852 23956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:57.391865 23956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:57.393391 23956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:43793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:43793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:57.393601 23956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:57.393778 23956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:57.398757 23961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:57.398752 23964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:57.398761 23962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:57.398878 23956 server_base.cc:1034] running on GCE node
I20260925 02:33:57.399235 23956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:57.399544 23956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:57.400722 23956 hybrid_clock.cc:648] HybridClock initialized: now 1790303637400707 us; error 44 us; skew 500 ppm
Sep 25 02:33:57 dist-test-slave-q8mg krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303637, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:57.403481 23956 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:33:57.404018 23956 webserver.cc:460] Webserver started at http://127.10.13.62:40035/ using document root <none> and password file <none>
I20260925 02:33:57.404254 23956 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:57.404325 23956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:57.404482 23956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:33:57.405756 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "6cefce72c33a466fbb9adede71ee4693"
format_stamp: "Formatted at 2026-09-25 02:33:57 on dist-test-slave-q8mg"
server_key: "2103cb80b9f4a059099d0d2ebe41bd2b"
server_key_iv: "1f624db8b47b79fe0ff5a8703e6af51e"
server_key_version: "encryptionkey@0"
I20260925 02:33:57.406167 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "6cefce72c33a466fbb9adede71ee4693"
format_stamp: "Formatted at 2026-09-25 02:33:57 on dist-test-slave-q8mg"
server_key: "2103cb80b9f4a059099d0d2ebe41bd2b"
server_key_iv: "1f624db8b47b79fe0ff5a8703e6af51e"
server_key_version: "encryptionkey@0"
I20260925 02:33:57.407629 23956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:33:57.408445 23971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:57.408650 23956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:57.408761 23956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "6cefce72c33a466fbb9adede71ee4693"
format_stamp: "Formatted at 2026-09-25 02:33:57 on dist-test-slave-q8mg"
server_key: "2103cb80b9f4a059099d0d2ebe41bd2b"
server_key_iv: "1f624db8b47b79fe0ff5a8703e6af51e"
server_key_version: "encryptionkey@0"
I20260925 02:33:57.408847 23956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:57.423974 23956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:57.426515 23956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:57.426690 23956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:57.431083 23956 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:43793
I20260925 02:33:57.431121 24023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:43793 every 8 connection(s)
I20260925 02:33:57.431540 23956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:33:57.432436 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:57.434942 24024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693: Bootstrap starting.
I20260925 02:33:57.435575 24024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:57.435890 24024 log.cc:824] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:57.436647 24024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693: No bootstrap required, opened a new log
I20260925 02:33:57.438331 24024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cefce72c33a466fbb9adede71ee4693" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43793 } }
I20260925 02:33:57.438484 24024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:57.438545 24024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cefce72c33a466fbb9adede71ee4693, State: Initialized, Role: FOLLOWER
I20260925 02:33:57.438697 24024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cefce72c33a466fbb9adede71ee4693" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43793 } }
I20260925 02:33:57.438762 24024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:33:57.438804 24024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:33:57.438858 24024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:57.439157 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 23956
I20260925 02:33:57.439250 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:33:57.439502 10292 external_mini_cluster.cc:1428] Setting key 0b29e1aa93de8a7323b72704946b9701
I20260925 02:33:57.439580 24024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cefce72c33a466fbb9adede71ee4693" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43793 } }
I20260925 02:33:57.439716 24024 leader_election.cc:302] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cefce72c33a466fbb9adede71ee4693; no voters:
I20260925 02:33:57.439927 24024 leader_election.cc:288] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:33:57.440222 24024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:33:57.440228 24027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:33:57.440518 24027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [term 1 LEADER]: Becoming Leader. State: Replica: 6cefce72c33a466fbb9adede71ee4693, State: Running, Role: LEADER
I20260925 02:33:57.440670 24027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cefce72c33a466fbb9adede71ee4693" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43793 } }
I20260925 02:33:57.441128 24029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cefce72c33a466fbb9adede71ee4693" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cefce72c33a466fbb9adede71ee4693" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43793 } } }
I20260925 02:33:57.441216 24029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:57.441397 24031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cefce72c33a466fbb9adede71ee4693. Latest consensus state: current_term: 1 leader_uuid: "6cefce72c33a466fbb9adede71ee4693" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cefce72c33a466fbb9adede71ee4693" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43793 } } }
I20260925 02:33:57.441471 24031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693 [sys.catalog]: This master's current role is: LEADER
W20260925 02:33:57.442490 24045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:33:57.442553 24045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:33:57.442600 24038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:33:57.442759 24038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:33:57.444500 24038 catalog_manager.cc:1347] Generated new cluster ID: 664f6c559db7412ca56cc46bc4cb7418
I20260925 02:33:57.444558 24038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:33:57 dist-test-slave-q8mg krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:57.452148 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:57.440015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.10.13.62:43793)
0925 02:33:57.440220 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:57.440231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:57.440273 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:33:57.440407 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:57.440414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:57.440481 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:57.441547 (+ 1066us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:57.441662 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:57.443100 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:57.444144 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:57.444673 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:57.445113 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:57.447653 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:57.447658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:57.447658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:57.447671 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:57.450326 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:57.450774 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:57.450778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:57.450782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:57.450920 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:57.451157 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:57.451157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:57.451158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:57.451470 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:57.451628 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:57.451762 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 02:33:57.451911 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260925 02:33:57.453348 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:33:57.461155 24038 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:33:57.461810 24038 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:33:57.464792 24038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cefce72c33a466fbb9adede71ee4693: Generated new TSK 0
I20260925 02:33:57.465022 24038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:33:57.507997 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 23956
2026-09-25T02:33:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1444 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:33:57 dist-test-slave-q8mg krb5kdc[24051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:33:57 dist-test-slave-q8mg krb5kdc[24051](info): set up 2 sockets
Sep 25 02:33:57 dist-test-slave-q8mg krb5kdc[24051](info): commencing operation
krb5kdc: starting...
W20260925 02:33:58.543560 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:33:58 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:33:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:33:58.668423 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34117
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:58.752008 24064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:58.752225 24064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:58.752247 24064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:58.753376 24064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:58.753413 24064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:58.753427 24064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:58.753439 24064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:58.753451 24064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:58.754879 24064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:58.755084 24064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:58.755254 24064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:58.760113 24069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:58.760106 24072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:58.760106 24070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:58.760499 24064 server_base.cc:1034] running on GCE node
I20260925 02:33:58.760779 24064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:58.761162 24064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:58.762337 24064 hybrid_clock.cc:648] HybridClock initialized: now 1790303638762323 us; error 51 us; skew 500 ppm
Sep 25 02:33:58 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303638, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:58.765079 24064 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:33:58.765483 24064 webserver.cc:460] Webserver started at http://127.10.13.62:38375/ using document root <none> and password file <none>
I20260925 02:33:58.765713 24064 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:58.765764 24064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:58.765924 24064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:33:58.767009 24064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "4ec81539c78043feac130f5193a977f9"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "cabe3332eacef4a08abd124c5495c0d9"
server_key_iv: "1f8ab7dab54e984299c627c57543c529"
server_key_version: "encryptionkey@0"
I20260925 02:33:58.767373 24064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "4ec81539c78043feac130f5193a977f9"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "cabe3332eacef4a08abd124c5495c0d9"
server_key_iv: "1f8ab7dab54e984299c627c57543c529"
server_key_version: "encryptionkey@0"
I20260925 02:33:58.768780 24064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 02:33:58.769721 24079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:58.769979 24064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:58.770078 24064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "4ec81539c78043feac130f5193a977f9"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "cabe3332eacef4a08abd124c5495c0d9"
server_key_iv: "1f8ab7dab54e984299c627c57543c529"
server_key_version: "encryptionkey@0"
I20260925 02:33:58.770149 24064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:58.782732 24064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:58.785056 24064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:58.785211 24064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:58.789196 24064 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:34117
I20260925 02:33:58.789353 24131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:34117 every 8 connection(s)
I20260925 02:33:58.789637 24064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:33:58.790706 24132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:58.793018 24132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Bootstrap starting.
I20260925 02:33:58.793571 24132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:58.793818 24132 log.cc:824] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:58.794056 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 24064
I20260925 02:33:58.794204 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:33:58.794446 10292 external_mini_cluster.cc:1428] Setting key e0941918c0e4de8aa09738667ebfeaf3
I20260925 02:33:58.794730 24132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: No bootstrap required, opened a new log
I20260925 02:33:58.796494 24132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:33:58.796650 24132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:58.796713 24132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec81539c78043feac130f5193a977f9, State: Initialized, Role: FOLLOWER
I20260925 02:33:58.796923 24132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:33:58.797022 24132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:33:58.797067 24132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:33:58.797118 24132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:58.797961 24132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:33:58.798084 24132 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec81539c78043feac130f5193a977f9; no voters:
I20260925 02:33:58.798275 24132 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:33:58.798297 24137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:33:58.798422 24137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 1 LEADER]: Becoming Leader. State: Replica: 4ec81539c78043feac130f5193a977f9, State: Running, Role: LEADER
I20260925 02:33:58.798560 24132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:33:58.798573 24137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:33:58.799021 24138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ec81539c78043feac130f5193a977f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } } }
I20260925 02:33:58.799034 24139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec81539c78043feac130f5193a977f9. Latest consensus state: current_term: 1 leader_uuid: "4ec81539c78043feac130f5193a977f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } } }
I20260925 02:33:58.799223 24139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:58.799436 24138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:58.799467 24146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:33:58.800129 24146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:33:58 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:58.801926 24146 catalog_manager.cc:1347] Generated new cluster ID: 1d7d77b88ec74ed49b1992e46070c548
I20260925 02:33:58.805492 24146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:33:58.807093 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:58.794983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.10.13.62:34117)
0925 02:33:58.795187 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:58.795197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:58.795234 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:33:58.796500 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:58.796509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:58.796564 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:58.796732 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:58.797529 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:58.798667 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:58.799791 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:58.800398 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:58.800511 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:58.803306 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:58.803311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:58.803311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:58.803325 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:58.805192 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:58.805824 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:58.805827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:58.805827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:58.805968 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:58.806192 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:58.806193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:58.806194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:58.806494 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:58.806653 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:58.806750 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:33:58.806901 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260925 02:33:58.829525 24146 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:33:58.830173 24146 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:33:58.842008 24146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Generated new TSK 0
I20260925 02:33:58.842257 24146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:33:58.876701 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34117
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:58.964727 24158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:58.965037 24158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:58.965070 24158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:58.966488 24158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:58.966535 24158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:58.966563 24158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:33:58.968102 24158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:58.968387 24158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:58.968659 24158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:58.974043 24164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:58.974108 24166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:58.973994 24163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:58.974242 24158 server_base.cc:1034] running on GCE node
I20260925 02:33:58.974501 24158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:58.974768 24158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:58.975941 24158 hybrid_clock.cc:648] HybridClock initialized: now 1790303638975921 us; error 40 us; skew 500 ppm
Sep 25 02:33:58 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303638, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:58.978575 24158 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:33:58.979102 24158 webserver.cc:460] Webserver started at http://127.10.13.1:33713/ using document root <none> and password file <none>
I20260925 02:33:58.979354 24158 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:58.979404 24158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:58.979578 24158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:33:58.980623 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "cb46adca088148b39444024e91407f72"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "0ff7224219835cf9889cf60c076d1841"
server_key_iv: "e47887348cd5cf49b12f537e66755b87"
server_key_version: "encryptionkey@0"
I20260925 02:33:58.981017 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "cb46adca088148b39444024e91407f72"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "0ff7224219835cf9889cf60c076d1841"
server_key_iv: "e47887348cd5cf49b12f537e66755b87"
server_key_version: "encryptionkey@0"
I20260925 02:33:58.982376 24158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:33:58.983203 24173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:58.983402 24158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:33:58.983510 24158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "cb46adca088148b39444024e91407f72"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "0ff7224219835cf9889cf60c076d1841"
server_key_iv: "e47887348cd5cf49b12f537e66755b87"
server_key_version: "encryptionkey@0"
I20260925 02:33:58.983585 24158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:58.995815 24158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:58.997896 24158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:58.998059 24158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:58.998293 24158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:58.998646 24158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:58.998708 24158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:58.998751 24158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:58.998778 24158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.005553 24158 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:32889
I20260925 02:33:59.005684 24286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:32889 every 8 connection(s)
I20260925 02:33:59.006227 24158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:33:59.012405 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 24158
I20260925 02:33:59.012531 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:33:59.012730 10292 external_mini_cluster.cc:1428] Setting key 25dd086833a976d3a2b6dc262d47326b
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303638, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:59.017647 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.007111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:34045 (local address 127.10.13.62:34117)
0925 02:33:59.007157 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.007159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.007430 (+ 271us) server_negotiation.cc:407] Connection header received
0925 02:33:59.008209 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.008211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.008230 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.008299 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:59.009524 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.010119 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.011332 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.011576 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.011655 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.014414 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:59.014415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:59.014416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:59.014423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:59.016151 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.016543 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.016545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.016545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.016612 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.016851 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.016852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.016852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.017060 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:59.017133 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.017390 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.017502 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:33:59.017737 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.007075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:34045)
0925 02:33:59.007255 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.007326 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.008052 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.008375 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.008379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.009127 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:59.009427 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.009435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.010223 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.010224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.011237 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.011240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.011664 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.011665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.011810 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.012377 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:59.012388 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:59.014319 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:59.016235 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.016241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.016243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.016466 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.016681 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.016682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.016682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.016786 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.017128 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:59.017134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:59.017252 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.017337 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.017456 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260925 02:33:59.018213 24287 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34117
I20260925 02:33:59.018309 24287 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:59.018501 24287 heartbeater.cc:511] Master 127.10.13.62:34117 requested a full tablet report, sending...
I20260925 02:33:59.019044 24096 ts_manager.cc:194] Registered new tserver with Master: cb46adca088148b39444024e91407f72 (127.10.13.1:32889)
I20260925 02:33:59.019654 24096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:34045
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:33:59.040431 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34117
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:59.130590 24292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:59.130838 24292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:59.130882 24292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:59.132081 24292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:59.132148 24292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:59.132193 24292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:33:59.133755 24292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:59.134050 24292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:59.134294 24292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:33:59.139350 24292 server_base.cc:1034] running on GCE node
W20260925 02:33:59.139278 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:59.139308 24298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:59.139278 24300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.139765 24292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:59.140059 24292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:59.141206 24292 hybrid_clock.cc:648] HybridClock initialized: now 1790303639141184 us; error 39 us; skew 500 ppm
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:59.143707 24292 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:33:59.144097 24292 webserver.cc:460] Webserver started at http://127.10.13.2:37345/ using document root <none> and password file <none>
I20260925 02:33:59.144294 24292 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:59.144336 24292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:59.144423 24292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:33:59.145519 24292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "56ae07361ad541388839d216ecdcee86"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "4a7821dbb94866be7cd6d93b45becd68"
server_key_iv: "b5319e13e19471ac83b19d2daa8bacf0"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.145808 24292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "56ae07361ad541388839d216ecdcee86"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "4a7821dbb94866be7cd6d93b45becd68"
server_key_iv: "b5319e13e19471ac83b19d2daa8bacf0"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.147118 24292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:33:59.147861 24307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.148061 24292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:59.148169 24292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "56ae07361ad541388839d216ecdcee86"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "4a7821dbb94866be7cd6d93b45becd68"
server_key_iv: "b5319e13e19471ac83b19d2daa8bacf0"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.148252 24292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:59.162035 24292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:59.164358 24292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:59.164523 24292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:59.164767 24292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:59.165112 24292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:59.165165 24292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.165199 24292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:59.165225 24292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.171300 24292 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:36571
I20260925 02:33:59.171346 24420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:36571 every 8 connection(s)
I20260925 02:33:59.171662 24292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:33:59.175787 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 24292
I20260925 02:33:59.175928 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:33:59.176149 10292 external_mini_cluster.cc:1428] Setting key 60520bf193624c9456fcf3116f94e742
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:59.182315 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.172410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:35003 (local address 127.10.13.62:34117)
0925 02:33:59.172474 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.172476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.172709 (+ 233us) server_negotiation.cc:407] Connection header received
0925 02:33:59.173321 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.173323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.173336 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.173438 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:59.174475 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.175013 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.176151 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.176447 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.176527 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.179158 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:59.179159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:59.179160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:59.179168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:59.180621 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.181233 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.181233 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.181234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.181289 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.181609 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.181610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.181610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.181729 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:59.181779 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.182168 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.182246 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:33:59.182650 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.172365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:35003)
0925 02:33:59.172571 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.172615 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.173182 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.173562 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.173565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.174158 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:59.174360 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.174365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.175113 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.175114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.176069 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.176072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.176742 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.176743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.176959 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.177535 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:59.177544 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:59.179058 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:59.180725 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.180734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.180737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.181153 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.181365 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.181366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.181366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.181535 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.181798 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:59.181807 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:59.182006 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.182129 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.182330 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":120,"threads_started":1}
I20260925 02:33:59.183277 24421 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34117
I20260925 02:33:59.183400 24421 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:59.183607 24421 heartbeater.cc:511] Master 127.10.13.62:34117 requested a full tablet report, sending...
I20260925 02:33:59.184088 24096 ts_manager.cc:194] Registered new tserver with Master: 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571)
I20260925 02:33:59.184473 24096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:35003
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:33:59.204133 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34117
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:59.287091 24426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:59.287325 24426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:59.287362 24426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:59.288506 24426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:59.288574 24426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:59.288620 24426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:33:59.290275 24426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:59.290573 24426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:59.290817 24426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:59.296252 24431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:59.296239 24434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.296344 24426 server_base.cc:1034] running on GCE node
W20260925 02:33:59.296244 24432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.296700 24426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:59.296994 24426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:59.298187 24426 hybrid_clock.cc:648] HybridClock initialized: now 1790303639298174 us; error 32 us; skew 500 ppm
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:59.301117 24426 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:33:59.301601 24426 webserver.cc:460] Webserver started at http://127.10.13.3:33333/ using document root <none> and password file <none>
I20260925 02:33:59.301859 24426 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:59.301926 24426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:59.302080 24426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:33:59.303139 24426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "688256133d6245509d835c7b5c15b21f"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "68e3a1a8a74229ac5e0a9bd0673e5a58"
server_key_iv: "f07283e4aaf8b3e23d6826761d1028f5"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.303511 24426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "688256133d6245509d835c7b5c15b21f"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "68e3a1a8a74229ac5e0a9bd0673e5a58"
server_key_iv: "f07283e4aaf8b3e23d6826761d1028f5"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.304838 24426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:33:59.305692 24441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.305858 24426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:33:59.305961 24426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "688256133d6245509d835c7b5c15b21f"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "68e3a1a8a74229ac5e0a9bd0673e5a58"
server_key_iv: "f07283e4aaf8b3e23d6826761d1028f5"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.306048 24426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:59.329123 24426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:59.331591 24426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:59.331769 24426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:59.332024 24426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:59.332353 24426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:59.332408 24426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.332453 24426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:59.332485 24426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.338761 24426 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:34853
I20260925 02:33:59.338801 24554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:34853 every 8 connection(s)
I20260925 02:33:59.339143 24426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:33:59.339491 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 24426
I20260925 02:33:59.339581 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:33:59.339777 10292 external_mini_cluster.cc:1428] Setting key 42c98b828d6803867420b1fa4d147072
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:59.349056 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.339939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:41203 (local address 127.10.13.62:34117)
0925 02:33:59.340003 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.340006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.340205 (+ 199us) server_negotiation.cc:407] Connection header received
0925 02:33:59.340961 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.340967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.340979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.341019 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:59.341979 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.342521 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.343561 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.343739 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.343794 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.346237 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:59.346239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:59.346240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:59.346247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:59.347677 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.348107 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.348108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.348108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.348160 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.348430 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.348431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.348431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.348537 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:59.348590 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.348851 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.348980 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:33:59.349349 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.339893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:41203)
0925 02:33:59.340079 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.340131 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.340815 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.341079 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.341083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.341678 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:59.341890 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.341896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.342619 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.342620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.343484 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.343487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.343841 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.343843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.343995 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.344538 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:59.344547 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:59.346140 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:59.347764 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.347771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.347773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.348036 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.348243 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.348244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.348244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.348365 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.348597 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:59.348608 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:59.348735 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.348821 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.349070 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260925 02:33:59.349890 24555 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34117
I20260925 02:33:59.350001 24555 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:59.350196 24555 heartbeater.cc:511] Master 127.10.13.62:34117 requested a full tablet report, sending...
I20260925 02:33:59.350595 24096 ts_manager.cc:194] Registered new tserver with Master: 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853)
I20260925 02:33:59.350976 24096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:41203
I20260925 02:33:59.351797 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:33:59.359220 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.353349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.10.13.62:34117)
0925 02:33:59.353424 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.353426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.353543 (+ 117us) server_negotiation.cc:407] Connection header received
0925 02:33:59.353612 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.353613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.353623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.353694 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:59.354272 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.354740 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.355479 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.355675 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.355753 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.357001 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:59.357003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:59.357003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:59.357010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:59.358205 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.358515 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.358516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.358517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.358568 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.358745 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.358746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.358746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.358909 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:59.358988 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.359095 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.359155 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:33:59.360750 24096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:33:59.362474 24096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:33:59.369505 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.365551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.10.13.3:34853)
0925 02:33:59.365922 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.365927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.365936 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:33:59.365978 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.365980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.365995 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.366066 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:59.366183 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.366906 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.368489 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.369072 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.369171 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.369238 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.369372 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.369413 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":95,"threads_started":1}
I20260925 02:33:59.369488 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.365312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:34853 (local address 127.0.0.1:58690)
0925 02:33:59.365540 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.365550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.365594 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.366065 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.366066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.366068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:59.366125 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.366127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.367011 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.367013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.368387 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.368390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.369173 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.369174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.369255 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.369270 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.369341 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.369410 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260925 02:33:59.369599 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.365228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.10.13.1:32889)
0925 02:33:59.365622 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.365627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.365637 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:33:59.365678 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.365680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.365695 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.365769 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:59.365996 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.367148 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.368358 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.369129 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.369206 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.369269 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.369478 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.369524 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":80,"threads_started":1}
I20260925 02:33:59.369683 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.365090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36571 (local address 127.0.0.1:40572)
0925 02:33:59.365416 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.365428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.365483 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.365787 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.365791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.365794 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:59.365899 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.365902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.367259 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.367260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.368393 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.368394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.369319 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.369320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.369379 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.369381 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.369433 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.369478 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":27,"thread_start_us":100,"threads_started":1}
I20260925 02:33:59.369705 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.365304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.10.13.2:36571)
0925 02:33:59.365520 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.365525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.365535 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:33:59.365579 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.365582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.365600 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.365681 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:59.365987 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.367156 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.368472 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.369182 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.369241 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.369289 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.369567 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.369613 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
I20260925 02:33:59.369918 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.365088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:32889 (local address 127.0.0.1:37850)
0925 02:33:59.365246 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.365293 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.365365 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.365787 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.365791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.365796 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:59.365899 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.365901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.367269 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.367270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.368253 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.368255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.369309 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.369310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.369377 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.369380 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.369712 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.369873 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":24,"thread_start_us":64,"threads_started":1}
I20260925 02:33:59.370512 24355 tablet_service.cc:1467] Processing CreateTablet for tablet d490cb9e9a66490d8099c572c2514bd6 (DEFAULT_TABLE table=test-table [id=7bc3565ccef44208abdc31636644af19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:59.370788 24489 tablet_service.cc:1467] Processing CreateTablet for tablet d490cb9e9a66490d8099c572c2514bd6 (DEFAULT_TABLE table=test-table [id=7bc3565ccef44208abdc31636644af19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:59.370910 24355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d490cb9e9a66490d8099c572c2514bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:59.371070 24489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d490cb9e9a66490d8099c572c2514bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:59.373661 24572 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Bootstrap starting.
I20260925 02:33:59.374157 24573 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: Bootstrap starting.
I20260925 02:33:59.374531 24572 tablet_bootstrap.cc:654] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:59.374604 24221 tablet_service.cc:1467] Processing CreateTablet for tablet d490cb9e9a66490d8099c572c2514bd6 (DEFAULT_TABLE table=test-table [id=7bc3565ccef44208abdc31636644af19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:33:59.374922 24573 tablet_bootstrap.cc:654] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:59.374956 24221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d490cb9e9a66490d8099c572c2514bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:33:59.375145 24572 log.cc:824] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:59.375483 24573 log.cc:824] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:59.375982 24572 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: No bootstrap required, opened a new log
I20260925 02:33:59.376029 24572 ts_tablet_manager.cc:1397] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:33:59.376230 24573 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: No bootstrap required, opened a new log
I20260925 02:33:59.376294 24573 ts_tablet_manager.cc:1397] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:33:59.377442 24572 raft_consensus.cc:348] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.377559 24572 raft_consensus.cc:374] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:59.377580 24572 raft_consensus.cc:729] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ae07361ad541388839d216ecdcee86, State: Initialized, Role: FOLLOWER
I20260925 02:33:59.377702 24572 consensus_queue.cc:260] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.377992 24572 ts_tablet_manager.cc:1428] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:33:59.378000 24421 heartbeater.cc:503] Master 127.10.13.62:34117 was elected leader, sending a full tablet report...
I20260925 02:33:59.378379 24577 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Bootstrap starting.
I20260925 02:33:59.378389 24573 raft_consensus.cc:348] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.378554 24573 raft_consensus.cc:374] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:59.378592 24573 raft_consensus.cc:729] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688256133d6245509d835c7b5c15b21f, State: Initialized, Role: FOLLOWER
I20260925 02:33:59.378703 24573 consensus_queue.cc:260] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.378993 24573 ts_tablet_manager.cc:1428] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:33:59.379002 24555 heartbeater.cc:503] Master 127.10.13.62:34117 was elected leader, sending a full tablet report...
I20260925 02:33:59.379127 24577 tablet_bootstrap.cc:654] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Neither blocks nor log segments found. Creating new log.
I20260925 02:33:59.379822 24577 log.cc:824] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:59.380662 24577 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: No bootstrap required, opened a new log
I20260925 02:33:59.380750 24577 ts_tablet_manager.cc:1397] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:33:59.382141 24577 raft_consensus.cc:348] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.382292 24577 raft_consensus.cc:374] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:33:59.382337 24577 raft_consensus.cc:729] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb46adca088148b39444024e91407f72, State: Initialized, Role: FOLLOWER
I20260925 02:33:59.382455 24577 consensus_queue.cc:260] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.382750 24577 ts_tablet_manager.cc:1428] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:33:59.382795 24287 heartbeater.cc:503] Master 127.10.13.62:34117 was elected leader, sending a full tablet report...
I20260925 02:33:59.401103 24576 raft_consensus.cc:482] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:33:59.401237 24576 raft_consensus.cc:504] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.401575 24576 leader_election.cc:288] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb46adca088148b39444024e91407f72 (127.10.13.1:32889), 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853)
I20260925 02:33:59.404434 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.401874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:33105 (local address 127.10.13.1:32889)
0925 02:33:59.401979 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.401981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.401990 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:33:59.402024 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.402025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.402047 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.402113 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:59.402282 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.402756 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.403686 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.404072 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.404156 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.404169 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.404323 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.404360 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:33:59.404434 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.401754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:32889 (local address 127.10.13.2:33105)
0925 02:33:59.401817 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.401831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.401894 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.402121 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.402122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.402124 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:59.402202 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.402203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.402843 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.402844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.403584 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.403585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.404157 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.404158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.404247 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.404250 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.404317 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.404360 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 02:33:59.404709 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.401938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:59923 (local address 127.10.13.3:34853)
0925 02:33:59.402023 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.402025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.402185 (+ 160us) server_negotiation.cc:407] Connection header received
0925 02:33:59.402254 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.402255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.402269 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.402338 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:59.402529 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.402987 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.403905 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.404285 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.404371 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.404381 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.404582 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.404638 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:33:59.404950 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.401907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:34853 (local address 127.10.13.2:59923)
0925 02:33:59.402106 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.402118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.402199 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.402352 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.402353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.402354 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:59.402445 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.402446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.403070 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.403071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.403812 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.403814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.404373 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.404374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.404468 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.404472 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.404808 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.404855 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":148,"threads_started":1}
I20260925 02:33:59.404839 24241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d490cb9e9a66490d8099c572c2514bd6" candidate_uuid: "56ae07361ad541388839d216ecdcee86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb46adca088148b39444024e91407f72" is_pre_election: true
I20260925 02:33:59.405071 24241 raft_consensus.cc:2454] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ae07361ad541388839d216ecdcee86 in term 0.
I20260925 02:33:59.405301 24310 leader_election.cc:302] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ae07361ad541388839d216ecdcee86, cb46adca088148b39444024e91407f72; no voters:
I20260925 02:33:59.405308 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d490cb9e9a66490d8099c572c2514bd6" candidate_uuid: "56ae07361ad541388839d216ecdcee86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688256133d6245509d835c7b5c15b21f" is_pre_election: true
I20260925 02:33:59.405459 24509 raft_consensus.cc:2454] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ae07361ad541388839d216ecdcee86 in term 0.
I20260925 02:33:59.405457 24576 raft_consensus.cc:2781] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:33:59.405561 24576 raft_consensus.cc:482] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:33:59.405598 24576 raft_consensus.cc:3037] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:59.406317 24576 raft_consensus.cc:504] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.406469 24576 leader_election.cc:288] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [CANDIDATE]: Term 1 election: Requested vote from peers cb46adca088148b39444024e91407f72 (127.10.13.1:32889), 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853)
I20260925 02:33:59.406625 24241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d490cb9e9a66490d8099c572c2514bd6" candidate_uuid: "56ae07361ad541388839d216ecdcee86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb46adca088148b39444024e91407f72"
I20260925 02:33:59.406666 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d490cb9e9a66490d8099c572c2514bd6" candidate_uuid: "56ae07361ad541388839d216ecdcee86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688256133d6245509d835c7b5c15b21f"
I20260925 02:33:59.406713 24241 raft_consensus.cc:3037] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:59.406728 24509 raft_consensus.cc:3037] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:33:59.407644 24241 raft_consensus.cc:2454] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ae07361ad541388839d216ecdcee86 in term 1.
I20260925 02:33:59.407644 24509 raft_consensus.cc:2454] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ae07361ad541388839d216ecdcee86 in term 1.
I20260925 02:33:59.407887 24310 leader_election.cc:302] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ae07361ad541388839d216ecdcee86, 688256133d6245509d835c7b5c15b21f; no voters:
I20260925 02:33:59.408007 24576 raft_consensus.cc:2781] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:33:59.408215 24576 raft_consensus.cc:686] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 1 LEADER]: Becoming Leader. State: Replica: 56ae07361ad541388839d216ecdcee86, State: Running, Role: LEADER
I20260925 02:33:59.408346 24576 consensus_queue.cc:237] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.409372 24095 catalog_manager.cc:5495] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56ae07361ad541388839d216ecdcee86 (127.10.13.2). New cstate: current_term: 1 leader_uuid: "56ae07361ad541388839d216ecdcee86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } health_report { overall_health: HEALTHY } } }
W20260925 02:33:59.422256 24422 tablet.cc:2369] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:33:59.422685 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.420068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.10.13.2:36571)
0925 02:33:59.420137 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.420139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.420148 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:33:59.420191 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.420192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.420211 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.420272 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:33:59.420388 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.420847 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.421644 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.421828 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.421883 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.422019 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:33:59.422129 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:33:59.422191 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.422534 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.422554 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:33:59.424533 24509 raft_consensus.cc:1261] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 1 FOLLOWER]: Refusing update from remote peer 56ae07361ad541388839d216ecdcee86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:33:59.424532 24241 raft_consensus.cc:1261] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Refusing update from remote peer 56ae07361ad541388839d216ecdcee86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:33:59.424907 24576 consensus_queue.cc:1035] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:33:59.425060 24582 consensus_queue.cc:1035] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:33:59.429958 24095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:33:59.430056 24095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:33:59.430845 24095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Sending DeleteTablet for 3 replicas of tablet d490cb9e9a66490d8099c572c2514bd6
I20260925 02:33:59.431201 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 24158
I20260925 02:33:59.431193 24355 tablet_service.cc:1514] Processing DeleteTablet for tablet d490cb9e9a66490d8099c572c2514bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:59 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:40572
I20260925 02:33:59.431267 24221 tablet_service.cc:1514] Processing DeleteTablet for tablet d490cb9e9a66490d8099c572c2514bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:59 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:37850
I20260925 02:33:59.431780 24489 tablet_service.cc:1514] Processing DeleteTablet for tablet d490cb9e9a66490d8099c572c2514bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:59 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:58690
I20260925 02:33:59.431855 24594 tablet_replica.cc:330] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: stopping tablet replica
I20260925 02:33:59.431947 24594 raft_consensus.cc:2229] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:33:59.432051 24594 raft_consensus.cc:2258] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:33:59.432529 24594 ts_tablet_manager.cc:1905] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:33:59.434275 24594 ts_tablet_manager.cc:1918] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:33:59.434356 24594 log.cc:1196] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/d490cb9e9a66490d8099c572c2514bd6
I20260925 02:33:59.434695 24594 ts_tablet_manager.cc:1939] T d490cb9e9a66490d8099c572c2514bd6 P 56ae07361ad541388839d216ecdcee86: Deleting consensus metadata
I20260925 02:33:59.435146 24081 catalog_manager.cc:4841] TS 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571): tablet d490cb9e9a66490d8099c572c2514bd6 (table test-table [id=7bc3565ccef44208abdc31636644af19]) successfully deleted
W20260925 02:33:59.437250 24082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:32889 (error 108)
I20260925 02:33:59.437485 24595 tablet_replica.cc:330] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: stopping tablet replica
I20260925 02:33:59.437618 24595 raft_consensus.cc:2229] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:33:59.437767 24595 raft_consensus.cc:2258] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:33:59.437798 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.437678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:32889 (local address 127.0.0.1:37858)
0925 02:33:59.437727 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.437761 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:33:59.437915 24082 catalog_manager.cc:4584] TS cb46adca088148b39444024e91407f72 (127.10.13.1:32889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d490cb9e9a66490d8099c572c2514bd6: Network error: Client connection negotiation failed: client connection to 127.10.13.1:32889: connect: Connection refused (error 111)
I20260925 02:33:59.438086 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 24292
I20260925 02:33:59.439970 24595 ts_tablet_manager.cc:1905] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:33:59.441303 24595 ts_tablet_manager.cc:1918] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:33:59.441383 24595 log.cc:1196] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/d490cb9e9a66490d8099c572c2514bd6
I20260925 02:33:59.441648 24595 ts_tablet_manager.cc:1939] T d490cb9e9a66490d8099c572c2514bd6 P 688256133d6245509d835c7b5c15b21f: Deleting consensus metadata
I20260925 02:33:59.442179 24082 catalog_manager.cc:4841] TS 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853): tablet d490cb9e9a66490d8099c572c2514bd6 (table test-table [id=7bc3565ccef44208abdc31636644af19]) successfully deleted
I20260925 02:33:59.445219 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 24426
I20260925 02:33:59.449977 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 24064
I20260925 02:33:59.455648 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34117
--webserver_interface=127.10.13.62
--webserver_port=38375
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:33:59.539009 24597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:59.539219 24597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:59.539242 24597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:59.540320 24597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:33:59.540356 24597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:59.540370 24597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:59.540382 24597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:33:59.540395 24597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:33:59.541982 24597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=38375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:59.542198 24597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:59.542367 24597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:59.552193 24604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:59.552246 24606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:59.552243 24603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.552410 24597 server_base.cc:1034] running on GCE node
I20260925 02:33:59.552667 24597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:59.553028 24597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:59.554173 24597 hybrid_clock.cc:648] HybridClock initialized: now 1790303639554140 us; error 55 us; skew 500 ppm
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:59.560583 24597 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:33:59.561199 24597 webserver.cc:460] Webserver started at http://127.10.13.62:38375/ using document root <none> and password file <none>
I20260925 02:33:59.561524 24597 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:59.561628 24597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:59.563189 24597 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:33:59.563931 24613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.564131 24597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:59.564250 24597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "4ec81539c78043feac130f5193a977f9"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "cabe3332eacef4a08abd124c5495c0d9"
server_key_iv: "1f8ab7dab54e984299c627c57543c529"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.564688 24597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:59.587291 24597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:59.589610 24597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:59.589789 24597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:59.594203 24597 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:34117
I20260925 02:33:59.594241 24665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:34117 every 8 connection(s)
I20260925 02:33:59.594655 24597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:33:59.596312 24666 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:33:59.597149 24666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Bootstrap starting.
I20260925 02:33:59.601121 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 24597
I20260925 02:33:59.601485 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:32889
--local_ip_for_outbound_sockets=127.10.13.1
--tserver_master_addrs=127.10.13.62:34117
--webserver_port=33713
--webserver_interface=127.10.13.1
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:33:59.606729 24666 log.cc:824] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:59.608779 24666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:33:59.609154 24666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Bootstrap complete.
I20260925 02:33:59.611546 24666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:33:59.611778 24666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec81539c78043feac130f5193a977f9, State: Initialized, Role: FOLLOWER
I20260925 02:33:59.611954 24666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:33:59.612030 24666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:33:59.612097 24666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:33:59.612154 24666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:33:59.613334 24666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:33:59.613456 24666 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec81539c78043feac130f5193a977f9; no voters:
I20260925 02:33:59.613669 24666 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:33:59.613706 24670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:33:59.613837 24670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 2 LEADER]: Becoming Leader. State: Replica: 4ec81539c78043feac130f5193a977f9, State: Running, Role: LEADER
I20260925 02:33:59.613984 24666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:33:59.614244 24670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:33:59.614583 24672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec81539c78043feac130f5193a977f9. Latest consensus state: current_term: 2 leader_uuid: "4ec81539c78043feac130f5193a977f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } } }
I20260925 02:33:59.614682 24672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:59.614914 24671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ec81539c78043feac130f5193a977f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } } }
I20260925 02:33:59.615022 24671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:33:59.615373 24679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:33:59.616410 24679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:33:59.616515 24679 catalog_manager.cc:1259] Loaded cluster ID: 1d7d77b88ec74ed49b1992e46070c548
I20260925 02:33:59.616549 24679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:33:59.616822 24679 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 02:33:59.621506 24679 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 02:33:59.624167 24679 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:33:59.624380 24679 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Loaded TSK: 0
I20260925 02:33:59.624609 24679 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:33:59.686618 24668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:59.686853 24668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:59.686892 24668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:59.688057 24668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:59.688122 24668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:59.688169 24668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:33:59.689719 24668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:32889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=33713
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:59.689987 24668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:59.690231 24668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:59.695264 24694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:59.695291 24693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.695564 24668 server_base.cc:1034] running on GCE node
W20260925 02:33:59.695358 24696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.695845 24668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:59.696135 24668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:59.697290 24668 hybrid_clock.cc:648] HybridClock initialized: now 1790303639697278 us; error 30 us; skew 500 ppm
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:59.700024 24668 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:33:59.700569 24668 webserver.cc:460] Webserver started at http://127.10.13.1:33713/ using document root <none> and password file <none>
I20260925 02:33:59.700834 24668 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:59.700951 24668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:59.702767 24668 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:33:59.703590 24703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.703810 24668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:59.703919 24668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "cb46adca088148b39444024e91407f72"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "0ff7224219835cf9889cf60c076d1841"
server_key_iv: "e47887348cd5cf49b12f537e66755b87"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.704387 24668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:59.716403 24668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:59.718804 24668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:59.718976 24668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:59.719220 24668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:59.719693 24710 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:33:59.721688 24668 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:33:59.721753 24668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:33:59.721817 24668 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:33:59.722465 24668 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:33:59.722523 24668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:33:59.722597 24710 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Bootstrap starting.
I20260925 02:33:59.728144 24668 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:32889
I20260925 02:33:59.728678 24668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:33:59.729869 24817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:32889 every 8 connection(s)
I20260925 02:33:59.733862 24710 log.cc:824] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Log is configured to *not* fsync() on all Append() calls
I20260925 02:33:59.736603 24710 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:59.737001 24710 tablet_bootstrap.cc:492] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Bootstrap complete.
I20260925 02:33:59.737207 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 24668
I20260925 02:33:59.737483 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:36571
--local_ip_for_outbound_sockets=127.10.13.2
--tserver_master_addrs=127.10.13.62:34117
--webserver_port=37345
--webserver_interface=127.10.13.2
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:33:59.738276 24710 ts_tablet_manager.cc:1397] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260925 02:33:59.739530 24710 raft_consensus.cc:348] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.739856 24710 raft_consensus.cc:729] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb46adca088148b39444024e91407f72, State: Initialized, Role: FOLLOWER
I20260925 02:33:59.740044 24710 consensus_queue.cc:260] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } } peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } }
I20260925 02:33:59.740334 24710 ts_tablet_manager.cc:1428] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 02:33:59.741852 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.730575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:59109 (local address 127.10.13.62:34117)
0925 02:33:59.731397 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.731405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.731430 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:33:59.731541 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.731545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.731585 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.731726 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:59.732407 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.733372 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.734658 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.735055 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.735180 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.737732 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:59.737736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:59.737736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:59.737749 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:59.739740 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.740306 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.740310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.740311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.740429 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.740712 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.740713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.740713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.741049 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:59.741159 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.741520 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.741658 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":68,"threads_started":1}
I20260925 02:33:59.741860 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.730476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:59109)
0925 02:33:59.730767 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.730802 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.731238 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.731754 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.731758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.732065 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:59.732251 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.732256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.733511 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.733512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.734554 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.734557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.735169 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.735171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.735295 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.735832 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:59.735842 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:59.737593 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:59.739823 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.739829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.739831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.740198 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.740516 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.740516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.740516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.740616 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.741149 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:59.741156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:59.741323 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.741448 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.741629 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260925 02:33:59.742522 24818 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34117
I20260925 02:33:59.742621 24818 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:59.742791 24818 heartbeater.cc:511] Master 127.10.13.62:34117 requested a full tablet report, sending...
I20260925 02:33:59.743556 24630 ts_manager.cc:194] Registered new tserver with Master: cb46adca088148b39444024e91407f72 (127.10.13.1:32889)
I20260925 02:33:59.745287 24630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:59109
I20260925 02:33:59.746397 24818 heartbeater.cc:503] Master 127.10.13.62:34117 was elected leader, sending a full tablet report...
I20260925 02:33:59.756423 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.745098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:32889 (local address 127.0.0.1:37868)
0925 02:33:59.751861 (+ 6763us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.751907 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.752001 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.752241 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.752244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.752247 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:33:59.752320 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.752322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.753704 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.753705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.755085 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.755088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.756021 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.756022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.756148 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.756168 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.756269 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.756335 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6724,"thread_start_us":95,"threads_started":1}
I20260925 02:33:59.756421 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.745162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.10.13.1:32889)
0925 02:33:59.749003 (+ 3841us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.749008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.751999 (+ 2991us) server_negotiation.cc:407] Connection header received
0925 02:33:59.752106 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.752109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.752126 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.752221 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:33:59.752398 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.753614 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.755191 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.755924 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.756002 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.756075 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.756292 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.756332 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3806,"thread_start_us":75,"threads_started":1}
I20260925 02:33:59.757333 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet d490cb9e9a66490d8099c572c2514bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:33:59 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:37868
I20260925 02:33:59.758013 24827 tablet_replica.cc:330] stopping tablet replica
I20260925 02:33:59.758137 24827 raft_consensus.cc:2229] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:33:59.758210 24827 raft_consensus.cc:2258] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:33:59.758540 24827 ts_tablet_manager.cc:1905] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:33:59.761150 24827 ts_tablet_manager.cc:1918] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:33:59.761233 24827 log.cc:1196] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/d490cb9e9a66490d8099c572c2514bd6
I20260925 02:33:59.761461 24827 ts_tablet_manager.cc:1939] T d490cb9e9a66490d8099c572c2514bd6 P cb46adca088148b39444024e91407f72: Deleting consensus metadata
I20260925 02:33:59.761998 24616 catalog_manager.cc:4841] TS cb46adca088148b39444024e91407f72 (127.10.13.1:32889): tablet d490cb9e9a66490d8099c572c2514bd6 (table test-table [id=7bc3565ccef44208abdc31636644af19]) successfully deleted
W20260925 02:33:59.828732 24823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:59.829010 24823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:59.829037 24823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:59.830160 24823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:59.830194 24823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:59.830219 24823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:33:59.831621 24823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:36571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=37345
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:59.831818 24823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:59.831996 24823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:59.837109 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.837281 24823 server_base.cc:1034] running on GCE node
W20260925 02:33:59.837137 24832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:59.837113 24835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.837575 24823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:59.837796 24823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:59.838945 24823 hybrid_clock.cc:648] HybridClock initialized: now 1790303639838926 us; error 31 us; skew 500 ppm
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:59.846262 24823 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:33:59.846728 24823 webserver.cc:460] Webserver started at http://127.10.13.2:37345/ using document root <none> and password file <none>
I20260925 02:33:59.846972 24823 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:59.847059 24823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:59.848515 24823 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:33:59.849228 24842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.849419 24823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:59.849531 24823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "56ae07361ad541388839d216ecdcee86"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "4a7821dbb94866be7cd6d93b45becd68"
server_key_iv: "b5319e13e19471ac83b19d2daa8bacf0"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.849920 24823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:33:59.864792 24823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:33:59.866784 24823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:33:59.866957 24823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:33:59.867202 24823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:33:59.867514 24823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:33:59.867568 24823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.867614 24823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:33:59.867637 24823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.873597 24823 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:36571
I20260925 02:33:59.873643 24955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:36571 every 8 connection(s)
I20260925 02:33:59.874017 24823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:33:59.875717 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 24823
I20260925 02:33:59.876008 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:34853
--local_ip_for_outbound_sockets=127.10.13.3
--tserver_master_addrs=127.10.13.62:34117
--webserver_port=33333
--webserver_interface=127.10.13.3
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:33:59.884703 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.874730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:35861 (local address 127.10.13.62:34117)
0925 02:33:59.874815 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 02:33:59.874817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:33:59.875041 (+ 224us) server_negotiation.cc:407] Connection header received
0925 02:33:59.875590 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:33:59.875592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:33:59.875612 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:33:59.875695 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:33:59.876777 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.877589 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.878880 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.879094 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.879156 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.881749 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:33:59.881750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:33:59.881751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:33:59.881759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:33:59.883210 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.883678 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.883680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.883680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.883745 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:33:59.884013 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:33:59.884014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:33:59.884015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:33:59.884137 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:33:59.884211 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:33:59.884504 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:33:59.884607 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:33:59.884972 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:33:59.874697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:35861)
0925 02:33:59.874877 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:33:59.874918 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:33:59.875454 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:33:59.875760 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:33:59.875764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:33:59.876476 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:33:59.876676 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.876681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.877699 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.877701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.878788 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:33:59.878792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:33:59.879186 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:33:59.879188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:33:59.879342 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:33:59.879923 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:33:59.879934 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:33:59.881644 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:33:59.883344 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.883353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.883355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.883598 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.883820 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:33:59.883821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:33:59.883821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:33:59.883941 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:33:59.884215 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:33:59.884225 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:33:59.884370 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 02:33:59.884480 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:33:59.884645 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260925 02:33:59.885421 24956 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34117
I20260925 02:33:59.885520 24956 heartbeater.cc:464] Registering TS with master...
I20260925 02:33:59.885737 24956 heartbeater.cc:511] Master 127.10.13.62:34117 requested a full tablet report, sending...
I20260925 02:33:59.886162 24630 ts_manager.cc:194] Registered new tserver with Master: 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571)
I20260925 02:33:59.886560 24630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:35861
W20260925 02:33:59.960677 24959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:33:59.960964 24959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:33:59.961006 24959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:33:59.962261 24959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:33:59.962325 24959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:33:59.962368 24959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:33:59.963857 24959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:34853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=33333
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:33:59.964152 24959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:33:59.964425 24959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:33:59.969658 24965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.969731 24959 server_base.cc:1034] running on GCE node
W20260925 02:33:59.969622 24964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:33:59.969631 24967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:33:59.970127 24959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:33:59.970399 24959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:33:59.971555 24959 hybrid_clock.cc:648] HybridClock initialized: now 1790303639971545 us; error 33 us; skew 500 ppm
Sep 25 02:33:59 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:33:59.974160 24959 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:33:59.974627 24959 webserver.cc:460] Webserver started at http://127.10.13.3:33333/ using document root <none> and password file <none>
I20260925 02:33:59.974882 24959 fs_manager.cc:359] Metadata directory not provided
I20260925 02:33:59.974978 24959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:33:59.976336 24959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:33:59.976972 24974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:33:59.977145 24959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:33:59.977249 24959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "688256133d6245509d835c7b5c15b21f"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "68e3a1a8a74229ac5e0a9bd0673e5a58"
server_key_iv: "f07283e4aaf8b3e23d6826761d1028f5"
server_key_version: "encryptionkey@0"
I20260925 02:33:59.977629 24959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:00.022455 24959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:00.025543 24959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:00.025740 24959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:00.026023 24959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:00.026420 24959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:00.026484 24959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.026540 24959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:00.026568 24959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.034651 24959 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:34853
I20260925 02:34:00.034708 25087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:34853 every 8 connection(s)
I20260925 02:34:00.035231 24959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:34:00.042209 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 24959
Sep 25 02:34:00 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303639, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:34:00.047418 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.036280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:52561 (local address 127.10.13.62:34117)
0925 02:34:00.036348 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:00.036350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:00.036417 (+ 67us) server_negotiation.cc:407] Connection header received
0925 02:34:00.037286 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:00.037287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:00.037297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:00.037343 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:00.038738 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.039240 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.041017 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.041208 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.041266 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.044019 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:00.044021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:00.044021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:00.044031 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:00.045780 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:00.046274 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:00.046275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:00.046275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:00.046349 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:00.046699 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:00.046700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:00.046700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:00.046818 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:00.046899 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:00.047231 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 02:34:00.047324 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:34:00.047658 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.036068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:52561)
0925 02:34:00.036263 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.036321 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:00.037157 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:00.037427 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:00.037433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:00.038323 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:00.038645 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.038652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.039335 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.039337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.040863 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.040931 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.041303 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.041304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.041482 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.042144 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:00.042156 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:00.043922 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:00.045901 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:00.045910 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:00.045912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:00.046202 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:00.046425 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:00.046426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:00.046426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:00.046581 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:00.046892 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:00.046899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:00.047085 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 02:34:00.047194 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:34:00.047367 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260925 02:34:00.048231 25088 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34117
I20260925 02:34:00.048346 25088 heartbeater.cc:464] Registering TS with master...
I20260925 02:34:00.048568 25088 heartbeater.cc:511] Master 127.10.13.62:34117 requested a full tablet report, sending...
I20260925 02:34:00.049034 24630 ts_manager.cc:194] Registered new tserver with Master: 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853)
I20260925 02:34:00.049592 24630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:52561
I20260925 02:34:00.052166 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.042642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.10.13.62:34117)
0925 02:34:00.042770 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:00.042773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:00.042933 (+ 160us) server_negotiation.cc:407] Connection header received
0925 02:34:00.043018 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:00.043020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:00.043033 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:00.043108 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:00.043945 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.044559 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.046206 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.046500 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.046578 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.049238 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:00.049240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:00.049240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:00.049248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:00.050967 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:00.051349 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:00.051350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:00.051351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:00.051413 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:00.051626 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:00.051627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:00.051627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:00.051741 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:00.051930 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:00.051967 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 02:34:00.052061 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260925 02:34:00.053184 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:34:00.055747 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.053540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.10.13.62:34117)
0925 02:34:00.053575 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:00.053577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:00.053671 (+ 94us) server_negotiation.cc:407] Connection header received
0925 02:34:00.053732 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:00.053732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:00.053761 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:00.053832 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:34:00.053938 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.054394 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.055153 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.055332 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.055382 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.055514 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:34:00.055601 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:34:00.055657 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:00.055686 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 02:34:00.055702 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 02:34:00.056002 24630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:34:00.057845 24630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:34:00.060331 24743 tablet_service.cc:1467] Processing CreateTablet for tablet d8af5bb38ec74493993afecc2f9c0719 (DEFAULT_TABLE table=test-table [id=26f11baf0616419ba592b0297a6e8eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:00.060570 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8af5bb38ec74493993afecc2f9c0719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:00.062284 24710 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Bootstrap starting.
I20260925 02:34:00.062626 24710 tablet_bootstrap.cc:654] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:00.063194 24710 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: No bootstrap required, opened a new log
I20260925 02:34:00.063251 24710 ts_tablet_manager.cc:1397] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:34:00.063382 24710 raft_consensus.cc:348] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.063476 24710 raft_consensus.cc:374] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:00.063537 24710 raft_consensus.cc:729] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb46adca088148b39444024e91407f72, State: Initialized, Role: FOLLOWER
I20260925 02:34:00.063588 24710 consensus_queue.cc:260] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.063699 24710 ts_tablet_manager.cc:1428] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.064167 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.059920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:34853 (local address 127.0.0.1:58702)
0925 02:34:00.060202 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.060214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:00.060260 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:00.060661 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:00.060662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:00.060664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:00.060711 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.060712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.062050 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.062051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.062982 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.062986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.063946 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.063947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.064009 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.064014 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:34:00.064044 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 02:34:00.064091 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
I20260925 02:34:00.064464 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.060018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36571 (local address 127.0.0.1:40594)
0925 02:34:00.060206 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.060217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:00.060265 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:00.060816 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:00.060818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:00.060820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:00.060865 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.060888 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.061936 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.061937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.063049 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.063052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.064188 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.064189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.064275 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.064280 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:34:00.064334 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:34:00.064386 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":24,"thread_start_us":134,"threads_started":1}
I20260925 02:34:00.064523 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.060462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.10.13.2:36571)
0925 02:34:00.060642 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:00.060647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:00.060657 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:34:00.060702 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:00.060704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:00.060716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:00.060803 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:00.060983 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.061819 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.063145 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.064004 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.064068 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.064131 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:00.064408 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 02:34:00.064447 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260925 02:34:00.064505 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.060057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58702 (local address 127.10.13.3:34853)
0925 02:34:00.060476 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:00.060481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:00.060492 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:34:00.060532 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:00.060534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:00.060550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:00.060787 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:00.060894 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.061919 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.063091 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.063831 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.064291 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.064358 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:00.064385 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 02:34:00.064426 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":114,"threads_started":1}
I20260925 02:34:00.065251 25022 tablet_service.cc:1467] Processing CreateTablet for tablet d8af5bb38ec74493993afecc2f9c0719 (DEFAULT_TABLE table=test-table [id=26f11baf0616419ba592b0297a6e8eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:00.065243 24890 tablet_service.cc:1467] Processing CreateTablet for tablet d8af5bb38ec74493993afecc2f9c0719 (DEFAULT_TABLE table=test-table [id=26f11baf0616419ba592b0297a6e8eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:00.065579 24890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8af5bb38ec74493993afecc2f9c0719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:00.065601 25022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8af5bb38ec74493993afecc2f9c0719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:00.068630 25098 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Bootstrap starting.
I20260925 02:34:00.069031 25099 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Bootstrap starting.
I20260925 02:34:00.069389 25098 tablet_bootstrap.cc:654] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:00.069744 25099 tablet_bootstrap.cc:654] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:00.069963 25098 log.cc:824] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:00.070345 25099 log.cc:824] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:00.070776 25098 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: No bootstrap required, opened a new log
I20260925 02:34:00.070855 25098 ts_tablet_manager.cc:1397] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:34:00.071152 25099 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: No bootstrap required, opened a new log
I20260925 02:34:00.071238 25099 ts_tablet_manager.cc:1397] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:34:00.072314 25098 raft_consensus.cc:348] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.072458 25098 raft_consensus.cc:374] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:00.072484 25098 raft_consensus.cc:729] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ae07361ad541388839d216ecdcee86, State: Initialized, Role: FOLLOWER
I20260925 02:34:00.072624 25098 consensus_queue.cc:260] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.072732 25099 raft_consensus.cc:348] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.072971 24956 heartbeater.cc:503] Master 127.10.13.62:34117 was elected leader, sending a full tablet report...
I20260925 02:34:00.072979 25099 raft_consensus.cc:374] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:00.073033 25099 raft_consensus.cc:729] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688256133d6245509d835c7b5c15b21f, State: Initialized, Role: FOLLOWER
I20260925 02:34:00.073180 25099 consensus_queue.cc:260] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.073284 25098 ts_tablet_manager.cc:1428] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:34:00.073508 25099 ts_tablet_manager.cc:1428] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:34:00.073598 25088 heartbeater.cc:503] Master 127.10.13.62:34117 was elected leader, sending a full tablet report...
W20260925 02:34:00.124634 24957 tablet.cc:2369] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:34:00.230722 24819 tablet.cc:2369] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:34:00.276968 25103 raft_consensus.cc:482] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:34:00.277103 25103 raft_consensus.cc:504] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.277480 25103 leader_election.cc:288] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571), cb46adca088148b39444024e91407f72 (127.10.13.1:32889)
I20260925 02:34:00.281777 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.277820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:38463 (local address 127.10.13.1:32889)
0925 02:34:00.277972 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:00.277974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:00.277985 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:34:00.278021 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:00.278022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:00.278031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:00.278117 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:00.278316 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.279112 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.280744 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.281408 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.281470 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.281481 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:00.281655 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 02:34:00.281699 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260925 02:34:00.281777 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.277780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:54723 (local address 127.10.13.2:36571)
0925 02:34:00.277822 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:00.277824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:00.278126 (+ 302us) server_negotiation.cc:407] Connection header received
0925 02:34:00.278160 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:00.278162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:00.278173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:00.278222 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:00.278376 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.279127 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.280745 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.281387 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.281468 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.281481 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:00.281654 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 02:34:00.281696 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 02:34:00.281798 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.277683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:32889 (local address 127.10.13.3:38463)
0925 02:34:00.277741 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.277755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:00.277809 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:00.278137 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:00.278138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:00.278140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:00.278237 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.278239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.279205 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.279206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.280648 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.280652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.281483 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.281484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.281571 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.281573 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:34:00.281636 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:34:00.281692 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":102}
I20260925 02:34:00.281804 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.277701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36571 (local address 127.10.13.3:54723)
0925 02:34:00.278006 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.278015 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:00.278073 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:00.278243 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:00.278243 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:00.278245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:00.278304 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.278305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.279205 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.279206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.280648 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:00.280652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.281484 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.281484 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:00.281567 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.281572 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:34:00.281636 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:34:00.281692 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":192,"thread_start_us":134,"threads_started":1}
I20260925 02:34:00.282209 24772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8af5bb38ec74493993afecc2f9c0719" candidate_uuid: "688256133d6245509d835c7b5c15b21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb46adca088148b39444024e91407f72" is_pre_election: true
I20260925 02:34:00.282428 24772 raft_consensus.cc:2454] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 688256133d6245509d835c7b5c15b21f in term 0.
I20260925 02:34:00.282527 24910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8af5bb38ec74493993afecc2f9c0719" candidate_uuid: "688256133d6245509d835c7b5c15b21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ae07361ad541388839d216ecdcee86" is_pre_election: true
I20260925 02:34:00.282675 24910 raft_consensus.cc:2454] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 688256133d6245509d835c7b5c15b21f in term 0.
I20260925 02:34:00.282675 24977 leader_election.cc:302] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 688256133d6245509d835c7b5c15b21f, cb46adca088148b39444024e91407f72; no voters:
I20260925 02:34:00.282822 25103 raft_consensus.cc:2781] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:34:00.282881 25103 raft_consensus.cc:482] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:34:00.282917 25103 raft_consensus.cc:3037] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:00.283619 25103 raft_consensus.cc:504] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.283775 25103 leader_election.cc:288] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [CANDIDATE]: Term 1 election: Requested vote from peers 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571), cb46adca088148b39444024e91407f72 (127.10.13.1:32889)
I20260925 02:34:00.283954 24772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8af5bb38ec74493993afecc2f9c0719" candidate_uuid: "688256133d6245509d835c7b5c15b21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb46adca088148b39444024e91407f72"
I20260925 02:34:00.284015 24910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8af5bb38ec74493993afecc2f9c0719" candidate_uuid: "688256133d6245509d835c7b5c15b21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ae07361ad541388839d216ecdcee86"
I20260925 02:34:00.284053 24772 raft_consensus.cc:3037] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:00.284086 24910 raft_consensus.cc:3037] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:00.284770 24772 raft_consensus.cc:2454] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 688256133d6245509d835c7b5c15b21f in term 1.
I20260925 02:34:00.284826 24910 raft_consensus.cc:2454] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 688256133d6245509d835c7b5c15b21f in term 1.
I20260925 02:34:00.285001 24977 leader_election.cc:302] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 688256133d6245509d835c7b5c15b21f, cb46adca088148b39444024e91407f72; no voters:
I20260925 02:34:00.285126 25103 raft_consensus.cc:2781] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:34:00.285259 25103 raft_consensus.cc:686] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 1 LEADER]: Becoming Leader. State: Replica: 688256133d6245509d835c7b5c15b21f, State: Running, Role: LEADER
I20260925 02:34:00.285333 25103 consensus_queue.cc:237] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
W20260925 02:34:00.285843 25089 tablet.cc:2369] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:34:00.286223 24630 catalog_manager.cc:5495] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f reported cstate change: term changed from 0 to 1, leader changed from <none> to 688256133d6245509d835c7b5c15b21f (127.10.13.3). New cstate: current_term: 1 leader_uuid: "688256133d6245509d835c7b5c15b21f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } health_report { overall_health: UNKNOWN } } }
I20260925 02:34:00.291708 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.289332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.10.13.3:34853)
0925 02:34:00.289401 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:00.289403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:00.289415 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:34:00.289495 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:00.289496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:00.289515 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:00.289629 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:34:00.289724 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.290181 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.290904 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:00.291116 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:00.291194 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:00.291361 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:34:00.291465 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:34:00.291570 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:00.291590 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 02:34:00.291610 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:34:00.293594 24772 raft_consensus.cc:1261] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Refusing update from remote peer 688256133d6245509d835c7b5c15b21f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:34:00.293594 24910 raft_consensus.cc:1261] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 1 FOLLOWER]: Refusing update from remote peer 688256133d6245509d835c7b5c15b21f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:34:00.293917 25103 consensus_queue.cc:1035] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:34:00.294050 25106 consensus_queue.cc:1035] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:34:00.296435 25110 mvcc.cc:205] Tried to move back new op lower bound from 7333083710639230976 to 7333083710609485824. Current Snapshot: MvccSnapshot[applied={T|T < 7333083710639230976}]
I20260925 02:34:00.296496 25111 mvcc.cc:205] Tried to move back new op lower bound from 7333083710639230976 to 7333083710609485824. Current Snapshot: MvccSnapshot[applied={T|T < 7333083710639230976}]
I20260925 02:34:00.296209 25109 mvcc.cc:205] Tried to move back new op lower bound from 7333083710639230976 to 7333083710609485824. Current Snapshot: MvccSnapshot[applied={T|T < 7333083710639230976}]
I20260925 02:34:00.298792 24629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:34:00.298905 24629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:34:00.299661 24629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Sending DeleteTablet for 3 replicas of tablet d8af5bb38ec74493993afecc2f9c0719
I20260925 02:34:00.299988 24890 tablet_service.cc:1514] Processing DeleteTablet for tablet d8af5bb38ec74493993afecc2f9c0719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:00 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:40594
I20260925 02:34:00.300026 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 24668
I20260925 02:34:00.300081 25022 tablet_service.cc:1514] Processing DeleteTablet for tablet d8af5bb38ec74493993afecc2f9c0719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:00 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:58702
I20260925 02:34:00.301017 25120 tablet_replica.cc:330] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: stopping tablet replica
I20260925 02:34:00.301118 25120 raft_consensus.cc:2229] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:34:00.301247 25120 raft_consensus.cc:2258] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:34:00.301887 25119 tablet_replica.cc:330] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: stopping tablet replica
I20260925 02:34:00.302038 25119 raft_consensus.cc:2229] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:34:00.302186 25119 raft_consensus.cc:2258] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:34:00.302687 25119 ts_tablet_manager.cc:1905] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:34:00.303287 25120 ts_tablet_manager.cc:1905] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:34:00.304770 25119 ts_tablet_manager.cc:1918] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:34:00.304908 25119 log.cc:1196] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/d8af5bb38ec74493993afecc2f9c0719
I20260925 02:34:00.304946 25120 ts_tablet_manager.cc:1918] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:34:00.305017 25120 log.cc:1196] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/d8af5bb38ec74493993afecc2f9c0719
I20260925 02:34:00.305317 25120 ts_tablet_manager.cc:1939] T d8af5bb38ec74493993afecc2f9c0719 P 688256133d6245509d835c7b5c15b21f: Deleting consensus metadata
I20260925 02:34:00.305317 25119 ts_tablet_manager.cc:1939] T d8af5bb38ec74493993afecc2f9c0719 P 56ae07361ad541388839d216ecdcee86: Deleting consensus metadata
I20260925 02:34:00.305785 24615 catalog_manager.cc:4841] TS 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571): tablet d8af5bb38ec74493993afecc2f9c0719 (table test-table [id=26f11baf0616419ba592b0297a6e8eac]) successfully deleted
I20260925 02:34:00.305892 24616 catalog_manager.cc:4841] TS 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853): tablet d8af5bb38ec74493993afecc2f9c0719 (table test-table [id=26f11baf0616419ba592b0297a6e8eac]) successfully deleted
W20260925 02:34:00.309118 24616 connection.cc:535] client connection to 127.10.13.1:32889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 02:34:00.309230 24616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:34:00.309453 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 24823
I20260925 02:34:00.311712 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.310152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:32889 (local address 127.0.0.1:37872)
0925 02:34:00.311627 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.311674 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1451}
W20260925 02:34:00.311870 24616 catalog_manager.cc:4584] TS cb46adca088148b39444024e91407f72 (127.10.13.1:32889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8af5bb38ec74493993afecc2f9c0719: Network error: Client connection negotiation failed: client connection to 127.10.13.1:32889: connect: Connection refused (error 111)
I20260925 02:34:00.315207 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 24959
I20260925 02:34:00.320226 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 24597
I20260925 02:34:00.326061 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34117
--webserver_interface=127.10.13.62
--webserver_port=38375
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:00.408418 25122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:00.408623 25122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:00.408646 25122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:00.409823 25122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:34:00.409864 25122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:00.409879 25122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:00.409893 25122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:34:00.409904 25122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:34:00.411372 25122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=38375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:00.411587 25122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:00.411774 25122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:00.416807 25128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:00.416767 25127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:00.416764 25130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:00.416916 25122 server_base.cc:1034] running on GCE node
I20260925 02:34:00.417251 25122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:00.417549 25122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:00.418713 25122 hybrid_clock.cc:648] HybridClock initialized: now 1790303640418706 us; error 49 us; skew 500 ppm
Sep 25 02:34:00 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303640, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:00.421401 25122 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:34:00.421923 25122 webserver.cc:460] Webserver started at http://127.10.13.62:38375/ using document root <none> and password file <none>
I20260925 02:34:00.422191 25122 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:00.422264 25122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:00.423730 25122 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:34:00.424459 25137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.424659 25122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:34:00.424770 25122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "4ec81539c78043feac130f5193a977f9"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "cabe3332eacef4a08abd124c5495c0d9"
server_key_iv: "1f8ab7dab54e984299c627c57543c529"
server_key_version: "encryptionkey@0"
I20260925 02:34:00.425196 25122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:00.441447 25122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:00.443640 25122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:00.443833 25122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:00.448170 25122 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:34117
I20260925 02:34:00.448311 25189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:34117 every 8 connection(s)
I20260925 02:34:00.448645 25122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:34:00.450273 25190 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:34:00.450887 25190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Bootstrap starting.
I20260925 02:34:00.451108 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 25122
I20260925 02:34:00.451335 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:32889
--local_ip_for_outbound_sockets=127.10.13.1
--tserver_master_addrs=127.10.13.62:34117
--webserver_port=33713
--webserver_interface=127.10.13.1
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:34:00.460139 25190 log.cc:824] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:00.462550 25190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:34:00.462905 25190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9: Bootstrap complete.
I20260925 02:34:00.465375 25190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:34:00.465601 25190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec81539c78043feac130f5193a977f9, State: Initialized, Role: FOLLOWER
I20260925 02:34:00.465780 25190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:34:00.465838 25190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:34:00.465914 25190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:34:00.465970 25190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:34:00.466989 25190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:34:00.467079 25190 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec81539c78043feac130f5193a977f9; no voters:
I20260925 02:34:00.467303 25190 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:34:00.467341 25194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:34:00.467478 25194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [term 3 LEADER]: Becoming Leader. State: Replica: 4ec81539c78043feac130f5193a977f9, State: Running, Role: LEADER
I20260925 02:34:00.467592 25190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:34:00.467576 25194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } }
I20260925 02:34:00.467922 25195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec81539c78043feac130f5193a977f9. Latest consensus state: current_term: 3 leader_uuid: "4ec81539c78043feac130f5193a977f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } } }
I20260925 02:34:00.468010 25195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:34:00.467933 25196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4ec81539c78043feac130f5193a977f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec81539c78043feac130f5193a977f9" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34117 } } }
I20260925 02:34:00.468175 25196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec81539c78043feac130f5193a977f9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:34:00.468230 25199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:34:00.469197 25199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:34:00.469383 25199 catalog_manager.cc:1259] Loaded cluster ID: 1d7d77b88ec74ed49b1992e46070c548
I20260925 02:34:00.469655 25199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 02:34:00.469954 25199 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe3ea1089ad google::LogMessage::Fail() at ??:0
@ 0x7fe3ea10b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe3ea1084de google::LogMessage::Flush() at ??:0
@ 0x7fe3ea109a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe3f20290f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe3f20328b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe3ebfd60d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe3ebfcb954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe3ece2f6db start_thread at ??:0
@ 0x7fe3e8c0661f clone at ??:0
W20260925 02:34:00.550154 25192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:00.550357 25192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:00.550381 25192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:00.551566 25192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:00.551612 25192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:00.551640 25192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:34:00.553593 25192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:32889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=33713
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:00.553905 25192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:00.554186 25192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:00.560501 25227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:00.560679 25192 server_base.cc:1034] running on GCE node
W20260925 02:34:00.560514 25229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:00.560498 25226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:00.561065 25192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:00.561334 25192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:00.562502 25192 hybrid_clock.cc:648] HybridClock initialized: now 1790303640562474 us; error 45 us; skew 500 ppm
Sep 25 02:34:00 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303640, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:00.565079 25192 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:34:00.565512 25192 webserver.cc:460] Webserver started at http://127.10.13.1:33713/ using document root <none> and password file <none>
I20260925 02:34:00.565788 25192 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:00.565835 25192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:00.567263 25192 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:00.568009 25236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.568197 25192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:00.568308 25192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "cb46adca088148b39444024e91407f72"
format_stamp: "Formatted at 2026-09-25 02:33:58 on dist-test-slave-q8mg"
server_key: "0ff7224219835cf9889cf60c076d1841"
server_key_iv: "e47887348cd5cf49b12f537e66755b87"
server_key_version: "encryptionkey@0"
I20260925 02:34:00.568674 25192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:00.585132 25192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:00.587630 25192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:00.587813 25192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:00.588073 25192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:00.588536 25243 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:34:00.590574 25192 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:34:00.590643 25192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:34:00.590692 25192 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:34:00.591344 25192 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:34:00.591403 25192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:34:00.591488 25243 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Bootstrap starting.
I20260925 02:34:00.599014 25192 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:32889
I20260925 02:34:00.599104 25350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:32889 every 8 connection(s)
I20260925 02:34:00.599493 25192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:34:00.603302 25243 log.cc:824] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:00.606261 25243 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:34:00.606611 25243 tablet_bootstrap.cc:492] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Bootstrap complete.
I20260925 02:34:00.607676 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 25192
I20260925 02:34:00.607983 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:36571
--local_ip_for_outbound_sockets=127.10.13.2
--tserver_master_addrs=127.10.13.62:34117
--webserver_port=37345
--webserver_interface=127.10.13.2
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:34:00.607975 25243 ts_tablet_manager.cc:1397] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20260925 02:34:00.609529 25243 raft_consensus.cc:348] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.609928 25243 raft_consensus.cc:729] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb46adca088148b39444024e91407f72, State: Initialized, Role: FOLLOWER
I20260925 02:34:00.610132 25243 consensus_queue.cc:260] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:00.610452 25243 ts_tablet_manager.cc:1428] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260925 02:34:00.714910 25353 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:34:00.600335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:47073)
0925 02:34:00.600556 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.600596 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:00.601148 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:00.714687 (+113539us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":123,"threads_started":1}
W20260925 02:34:00.715250 25351 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:34117 (0 consecutive failures): Network error: Failed to ping master at 127.10.13.62:34117: Client connection negotiation failed: client connection to 127.10.13.62:34117: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:34:00.715651 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.715507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:41567)
0925 02:34:00.715567 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.715620 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 02:34:00.716082 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.715939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:59457)
0925 02:34:00.715986 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.716010 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:34:00.716240 25351 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:34:00.723418 25355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:00.723613 25355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:00.723636 25355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:00.724841 25355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:00.724943 25355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:00.724992 25355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:34:00.726487 25355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:36571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=37345
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:00.726732 25355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:00.726976 25355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:00.731946 25361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:00.731979 25362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:00.731979 25364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:00.732225 25355 server_base.cc:1034] running on GCE node
I20260925 02:34:00.732461 25355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:00.732713 25355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:00.733889 25355 hybrid_clock.cc:648] HybridClock initialized: now 1790303640733860 us; error 41 us; skew 500 ppm
Sep 25 02:34:00 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303640, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:00.736414 25355 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:34:00.736814 25355 webserver.cc:460] Webserver started at http://127.10.13.2:37345/ using document root <none> and password file <none>
I20260925 02:34:00.737102 25355 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:00.737154 25355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:00.738557 25355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:00.739161 25371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.739357 25355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:34:00.739466 25355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "56ae07361ad541388839d216ecdcee86"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "4a7821dbb94866be7cd6d93b45becd68"
server_key_iv: "b5319e13e19471ac83b19d2daa8bacf0"
server_key_version: "encryptionkey@0"
I20260925 02:34:00.739842 25355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:00.755679 25355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:00.758103 25355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:00.758277 25355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:00.758530 25355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:00.758859 25355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:00.758911 25355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.758951 25355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:00.758980 25355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.764979 25355 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:36571
I20260925 02:34:00.765053 25484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:36571 every 8 connection(s)
I20260925 02:34:00.765416 25355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:34:00.766558 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.766168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:37457)
0925 02:34:00.766323 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.766402 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
W20260925 02:34:00.766991 25485 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:34117 (0 consecutive failures): Network error: Failed to ping master at 127.10.13.62:34117: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
I20260925 02:34:00.767268 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.767179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:38923)
0925 02:34:00.767224 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.767251 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 02:34:00.767498 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.767449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:42863)
0925 02:34:00.767470 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.767487 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 02:34:00.767612 25485 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:34:00.774401 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 25355
I20260925 02:34:00.774652 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:34853
--local_ip_for_outbound_sockets=127.10.13.3
--tserver_master_addrs=127.10.13.62:34117
--webserver_port=33333
--webserver_interface=127.10.13.3
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:00.860235 25488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:00.860453 25488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:00.860483 25488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:00.861639 25488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:00.861678 25488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:00.861704 25488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:34:00.863154 25488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:34853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=33333
--tserver_master_addrs=127.10.13.62:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:00.863363 25488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:00.863557 25488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:34:00.868718 25488 server_base.cc:1034] running on GCE node
W20260925 02:34:00.868752 25494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:00.868726 25496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:00.868772 25493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:00.869122 25488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:00.869380 25488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:00.870582 25488 hybrid_clock.cc:648] HybridClock initialized: now 1790303640870565 us; error 27 us; skew 500 ppm
Sep 25 02:34:00 dist-test-slave-q8mg krb5kdc[24051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303640, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:00.873310 25488 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:34:00.873811 25488 webserver.cc:460] Webserver started at http://127.10.13.3:33333/ using document root <none> and password file <none>
I20260925 02:34:00.874092 25488 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:00.874166 25488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:00.875607 25488 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:00.876235 25503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.876418 25488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:34:00.876530 25488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "688256133d6245509d835c7b5c15b21f"
format_stamp: "Formatted at 2026-09-25 02:33:59 on dist-test-slave-q8mg"
server_key: "68e3a1a8a74229ac5e0a9bd0673e5a58"
server_key_iv: "f07283e4aaf8b3e23d6826761d1028f5"
server_key_version: "encryptionkey@0"
I20260925 02:34:00.876969 25488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:00.887629 25488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:00.889567 25488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:00.889727 25488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:00.889955 25488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:00.890272 25488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:00.890331 25488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.890373 25488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:00.890411 25488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:00.896292 25488 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:34853
I20260925 02:34:00.896411 25616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:34853 every 8 connection(s)
I20260925 02:34:00.896732 25488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:34:00.897792 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.897418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:33533)
0925 02:34:00.897567 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.897645 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260925 02:34:00.898267 25617 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:34117 (0 consecutive failures): Network error: Failed to ping master at 127.10.13.62:34117: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
I20260925 02:34:00.898571 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.898478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:49053)
0925 02:34:00.898523 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.898553 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 02:34:00.898809 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:00.898761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:60915)
0925 02:34:00.898781 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:00.898798 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 02:34:00.898921 25617 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:34:00.899739 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 25488
I20260925 02:34:01.716823 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:01.716602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:53205)
0925 02:34:01.716722 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:01.716755 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260925 02:34:01.768200 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:01.767993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:42025)
0925 02:34:01.768115 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:01.768149 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260925 02:34:01.899492 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:01.899270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:39975)
0925 02:34:01.899402 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:01.899439 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260925 02:34:02.175334 25624 raft_consensus.cc:482] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:34:02.175467 25624 raft_consensus.cc:504] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:02.175768 25624 leader_election.cc:288] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853), 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571)
Sep 25 02:34:02 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303640, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.2@KRBTEST.COM
Sep 25 02:34:02 dist-test-slave-q8mg krb5kdc[24051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303640, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.3@KRBTEST.COM
I20260925 02:34:02.187022 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:02.176111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:59661 (local address 127.10.13.3:34853)
0925 02:34:02.176261 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:02.176275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:02.176313 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:34:02.176778 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:02.176780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:02.176824 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:02.177003 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:02.177816 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:02.178683 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:02.179728 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:02.180166 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:02.180746 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:02.183182 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:02.183186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:02.183187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:02.183199 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:02.185287 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:02.185706 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:02.185708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:02.185709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:02.185792 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:02.186127 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:02.186128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:02.186129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:02.186388 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:02.186484 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:02.186764 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 02:34:02.186897 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260925 02:34:02.187014 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:02.175967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:34853 (local address 127.10.13.1:59661)
0925 02:34:02.176159 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:02.176170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:02.176226 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:02.177076 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:02.177080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:02.177495 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:02.177688 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:02.177694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:02.178816 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:02.178817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:02.179648 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:02.179651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:02.180319 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:02.180320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:02.180427 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:02.181152 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:02.181160 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:02.183077 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:02.185371 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:02.185378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:02.185380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:02.185626 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:02.185924 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:02.185925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:02.185926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:02.186043 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:02.186461 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:02.186469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:02.186597 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 02:34:02.186708 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:34:02.186917 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":71,"thread_start_us":73,"threads_started":1}
I20260925 02:34:02.187101 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:02.176161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:50221 (local address 127.10.13.2:36571)
0925 02:34:02.176360 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:02.176368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:02.176395 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:34:02.176815 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:02.176818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:02.176923 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:02.177082 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:02.177840 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:02.178671 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:02.179757 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:02.180188 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:02.180325 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:02.182856 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:02.182862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:02.182862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:02.182876 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:02.185287 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:02.185931 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:02.185935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:02.185935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:02.186047 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:02.186267 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:02.186268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:02.186269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:02.186513 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:02.186606 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:02.186809 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 02:34:02.186968 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260925 02:34:02.187132 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:02.175967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:36571 (local address 127.10.13.1:50221)
0925 02:34:02.176029 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:02.176043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:02.176122 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:02.177076 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:02.177084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:02.177493 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:02.177687 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:02.177694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:02.178780 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:02.178781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:02.179652 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:02.179655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:02.180298 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:02.180299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:02.180423 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:02.181211 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:02.181216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:02.182716 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:02.185704 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:02.185704 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:02.185705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:02.185834 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:02.186131 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:02.186132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:02.186133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:02.186194 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:02.186600 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:02.186601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:02.186657 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 02:34:02.186739 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:34:02.187000 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":184}
W20260925 02:34:02.187920 25238 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
W20260925 02:34:02.188347 25239 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
I20260925 02:34:02.188390 25239 leader_election.cc:302] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb46adca088148b39444024e91407f72; no voters: 56ae07361ad541388839d216ecdcee86, 688256133d6245509d835c7b5c15b21f
I20260925 02:34:02.188555 25624 raft_consensus.cc:2726] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:34:02.717646 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:02.717382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:36791)
0925 02:34:02.717549 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:02.717596 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260925 02:34:02.768981 25629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:02.768703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:59047)
0925 02:34:02.768840 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:02.768936 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260925 02:34:02.900235 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:02.900014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:42693)
0925 02:34:02.900156 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:02.900193 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260925 02:34:03.718482 25631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:03.718225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:49005)
0925 02:34:03.718378 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:03.718419 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260925 02:34:03.769709 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:03.769510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:40179)
0925 02:34:03.769630 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:03.769664 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260925 02:34:03.900985 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:03.900730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:56061)
0925 02:34:03.900901 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:03.900942 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260925 02:34:04.134779 25634 raft_consensus.cc:482] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:34:04.134876 25634 raft_consensus.cc:504] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:04.135067 25634 leader_election.cc:288] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853), 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571)
W20260925 02:34:04.135509 25239 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
W20260925 02:34:04.135596 25238 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
I20260925 02:34:04.135700 25238 leader_election.cc:302] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb46adca088148b39444024e91407f72; no voters: 56ae07361ad541388839d216ecdcee86, 688256133d6245509d835c7b5c15b21f
I20260925 02:34:04.135785 25634 raft_consensus.cc:2726] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:34:04.719337 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:04.719039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:48343)
0925 02:34:04.719213 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:04.719267 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260925 02:34:04.770406 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:04.770202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:50387)
0925 02:34:04.770321 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:04.770361 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260925 02:34:04.901726 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:04.901442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:42195)
0925 02:34:04.901645 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:04.901683 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":120,"threads_started":1}
I20260925 02:34:05.720220 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:05.719951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:35159)
0925 02:34:05.720123 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:05.720169 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260925 02:34:05.771100 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:05.770873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:33185)
0925 02:34:05.771017 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:05.771056 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260925 02:34:05.902603 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:05.902295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:36899)
0925 02:34:05.902466 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:05.902543 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260925 02:34:06.237808 25641 raft_consensus.cc:482] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:34:06.237921 25641 raft_consensus.cc:504] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:06.238114 25641 leader_election.cc:288] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853), 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571)
W20260925 02:34:06.238556 25238 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
W20260925 02:34:06.238668 25239 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
I20260925 02:34:06.238754 25239 leader_election.cc:302] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb46adca088148b39444024e91407f72; no voters: 56ae07361ad541388839d216ecdcee86, 688256133d6245509d835c7b5c15b21f
I20260925 02:34:06.238891 25641 raft_consensus.cc:2726] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:34:06.721027 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:06.720756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:56105)
0925 02:34:06.720927 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:06.720975 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260925 02:34:06.771823 25643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:06.771607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:40825)
0925 02:34:06.771733 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:06.771772 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260925 02:34:06.903352 25644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:06.903134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:33259)
0925 02:34:06.903262 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:06.903305 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260925 02:34:07.721858 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:07.721624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:45379)
0925 02:34:07.721773 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:07.721811 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260925 02:34:07.772488 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:07.772250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:60719)
0925 02:34:07.772398 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:07.772439 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260925 02:34:07.904181 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:07.903904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:37373)
0925 02:34:07.904095 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:07.904134 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
I20260925 02:34:08.722595 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:08.722365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:60603)
0925 02:34:08.722499 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:08.722539 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260925 02:34:08.773272 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:08.773031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:35665)
0925 02:34:08.773172 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:08.773214 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260925 02:34:08.904932 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:08.904662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:55977)
0925 02:34:08.904785 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:08.904826 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260925 02:34:08.969126 25651 raft_consensus.cc:482] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:34:08.969214 25651 raft_consensus.cc:504] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:08.969434 25651 leader_election.cc:288] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853), 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571)
W20260925 02:34:08.969911 25239 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
W20260925 02:34:08.970003 25238 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
I20260925 02:34:08.970037 25238 leader_election.cc:302] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb46adca088148b39444024e91407f72; no voters: 56ae07361ad541388839d216ecdcee86, 688256133d6245509d835c7b5c15b21f
I20260925 02:34:08.970122 25651 raft_consensus.cc:2726] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:34:09.723425 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:09.723180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:59141)
0925 02:34:09.723322 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:09.723364 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260925 02:34:09.774029 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:09.773745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:36319)
0925 02:34:09.773935 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:09.773981 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260925 02:34:09.905773 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:09.905425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:50819)
0925 02:34:09.905596 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:09.905697 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260925 02:34:10.724115 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:10.723911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:55689)
0925 02:34:10.724036 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:10.724068 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260925 02:34:10.774786 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:10.774528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:36867)
0925 02:34:10.774692 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:10.774739 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260925 02:34:10.906531 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:10.906289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:37809)
0925 02:34:10.906439 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:10.906482 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260925 02:34:11.724927 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:11.724662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:52343)
0925 02:34:11.724808 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:11.724857 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260925 02:34:11.775539 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:11.775326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:55941)
0925 02:34:11.775453 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:11.775494 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260925 02:34:11.907223 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:11.907010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:54073)
0925 02:34:11.907137 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:11.907173 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 02:34:12.725659 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:12.725417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:50317)
0925 02:34:12.725567 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:12.725609 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260925 02:34:12.776288 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:12.776060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:52373)
0925 02:34:12.776201 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:12.776239 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260925 02:34:12.907977 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:12.907738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:33277)
0925 02:34:12.907885 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:12.907928 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260925 02:34:13.726516 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:13.726227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:55723)
0925 02:34:13.726391 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:13.726468 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260925 02:34:13.777200 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:13.776970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:47595)
0925 02:34:13.777108 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:13.777151 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260925 02:34:13.908720 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:13.908491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:41697)
0925 02:34:13.908632 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:13.908671 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260925 02:34:14.540005 25667 raft_consensus.cc:482] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:34:14.540125 25667 raft_consensus.cc:504] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688256133d6245509d835c7b5c15b21f" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34853 } } peers { permanent_uuid: "56ae07361ad541388839d216ecdcee86" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 36571 } } peers { permanent_uuid: "cb46adca088148b39444024e91407f72" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 32889 } }
I20260925 02:34:14.540351 25667 leader_election.cc:288] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853), 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571)
W20260925 02:34:14.540758 25238 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56ae07361ad541388839d216ecdcee86 (127.10.13.2:36571): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
W20260925 02:34:14.540846 25239 leader_election.cc:341] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 688256133d6245509d835c7b5c15b21f (127.10.13.3:34853): Not found: Tablet not found: d8af5bb38ec74493993afecc2f9c0719
I20260925 02:34:14.540980 25239 leader_election.cc:302] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb46adca088148b39444024e91407f72; no voters: 56ae07361ad541388839d216ecdcee86, 688256133d6245509d835c7b5c15b21f
I20260925 02:34:14.541091 25667 raft_consensus.cc:2726] T d8af5bb38ec74493993afecc2f9c0719 P cb46adca088148b39444024e91407f72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:34:14.727259 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:14.727031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:60135)
0925 02:34:14.727166 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:14.727208 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260925 02:34:14.777941 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:14.777682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:45967)
0925 02:34:14.777841 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:14.777890 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260925 02:34:14.909469 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:14.909211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:57501)
0925 02:34:14.909375 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:14.909409 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260925 02:34:15.728055 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:15.727823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.1:40463)
0925 02:34:15.727969 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:15.728008 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260925 02:34:15.778699 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:15.778494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.2:41433)
0925 02:34:15.778614 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:15.778651 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260925 02:34:15.910112 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 25192
I20260925 02:34:15.910955 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:15.910163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34117 (local address 127.10.13.3:43897)
0925 02:34:15.910855 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:15.910896 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":82,"threads_started":1}
I20260925 02:34:15.915534 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 25355
I20260925 02:34:15.920207 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 25488
2026-09-25T02:34:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18413 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:34:15 dist-test-slave-q8mg krb5kdc[25676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:34:15 dist-test-slave-q8mg krb5kdc[25676](info): set up 2 sockets
Sep 25 02:34:15 dist-test-slave-q8mg krb5kdc[25676](info): commencing operation
krb5kdc: starting...
W20260925 02:34:17.003628 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:34:17.127799 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39519
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:17.212934 25689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:17.213172 25689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:17.213212 25689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:17.214399 25689 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:34:17.214461 25689 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:34:17.214483 25689 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:34:17.214499 25689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:34:17.214527 25689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:17.214558 25689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:17.214584 25689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:34:17.214617 25689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:34:17.216123 25689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:17.216419 25689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:17.216662 25689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:17.222160 25694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:17.222188 25697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:17.222199 25695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:17.222249 25689 server_base.cc:1034] running on GCE node
I20260925 02:34:17.222697 25689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:17.222994 25689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:17.224161 25689 hybrid_clock.cc:648] HybridClock initialized: now 1790303657224150 us; error 45 us; skew 500 ppm
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:17.226967 25689 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:34:17.227483 25689 webserver.cc:460] Webserver started at http://127.10.13.62:44385/ using document root <none> and password file <none>
I20260925 02:34:17.227742 25689 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:17.227814 25689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:17.227969 25689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:34:17.229112 25689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "e0ed57e3d44c453a9778129e3005c039"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "60f04478e590c5ec6a9de6066a740c1e"
server_key_iv: "93bbb3e765c3b6e35e7a206a8dd41431"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.229470 25689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "e0ed57e3d44c453a9778129e3005c039"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "60f04478e590c5ec6a9de6066a740c1e"
server_key_iv: "93bbb3e765c3b6e35e7a206a8dd41431"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.230840 25689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:34:17.231572 25704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.231750 25689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:34:17.231860 25689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "e0ed57e3d44c453a9778129e3005c039"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "60f04478e590c5ec6a9de6066a740c1e"
server_key_iv: "93bbb3e765c3b6e35e7a206a8dd41431"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.231940 25689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:17.257627 25689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:17.260128 25689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:17.260310 25689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:17.264569 25689 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:39519
I20260925 02:34:17.264611 25756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:39519 every 8 connection(s)
I20260925 02:34:17.265012 25689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:34:17.265955 25757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:17.268253 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Bootstrap starting.
I20260925 02:34:17.268899 25757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:17.269200 25757 log.cc:824] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:17.269884 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: No bootstrap required, opened a new log
I20260925 02:34:17.271518 25757 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:17.271658 25757 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:17.271680 25757 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ed57e3d44c453a9778129e3005c039, State: Initialized, Role: FOLLOWER
I20260925 02:34:17.271854 25757 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:17.271944 25757 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:34:17.271968 25757 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:34:17.272047 25757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:17.272578 25757 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:17.272707 25757 leader_election.cc:302] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ed57e3d44c453a9778129e3005c039; no voters:
I20260925 02:34:17.272925 25757 leader_election.cc:288] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:34:17.272931 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 25689
I20260925 02:34:17.272970 25760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:34:17.273046 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:34:17.273176 25760 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 1 LEADER]: Becoming Leader. State: Replica: e0ed57e3d44c453a9778129e3005c039, State: Running, Role: LEADER
I20260925 02:34:17.273226 25757 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:34:17.273281 10292 external_mini_cluster.cc:1428] Setting key 4ada6e52cfbaefc640b7cc2c405e2634
I20260925 02:34:17.273272 25760 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:17.274029 25761 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0ed57e3d44c453a9778129e3005c039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } } }
I20260925 02:34:17.274118 25761 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: This master's current role is: LEADER
I20260925 02:34:17.274139 25762 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ed57e3d44c453a9778129e3005c039. Latest consensus state: current_term: 1 leader_uuid: "e0ed57e3d44c453a9778129e3005c039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } } }
I20260925 02:34:17.274274 25762 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: This master's current role is: LEADER
I20260925 02:34:17.274623 25771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:34:17.275267 25771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:34:17.277148 25771 catalog_manager.cc:1347] Generated new cluster ID: 0dd16d79f7f346669130e09ad68b9354
I20260925 02:34:17.277204 25771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:34:17.286124 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.273985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.10.13.62:39519)
0925 02:34:17.274324 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.274335 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.274424 (+ 89us) server_negotiation.cc:407] Connection header received
0925 02:34:17.274606 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.274614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.274687 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.275702 (+ 1015us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:17.275800 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.277192 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.278346 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.278961 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.279224 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.282220 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:17.282227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:17.282227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:17.282244 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:17.284361 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.284771 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.284775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.284775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.284920 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.285133 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.285134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.285135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.285490 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:17.285655 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.285740 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.285886 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":105,"threads_started":1}
I20260925 02:34:17.304497 25771 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:34:17.305302 25771 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:34:17.310964 25771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Generated new TSK 0
I20260925 02:34:17.311204 25771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:34:17.345198 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39519
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:17.429442 25783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:17.429693 25783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:17.429738 25783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:17.430959 25783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:17.431005 25783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:17.431061 25783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:34:17.432577 25783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:17.432884 25783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:17.433138 25783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:17.438661 25789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:17.438637 25791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:17.438685 25783 server_base.cc:1034] running on GCE node
W20260925 02:34:17.438637 25788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:17.439234 25783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:17.439571 25783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:17.440765 25783 hybrid_clock.cc:648] HybridClock initialized: now 1790303657440753 us; error 32 us; skew 500 ppm
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:17.443483 25783 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:34:17.443979 25783 webserver.cc:460] Webserver started at http://127.10.13.1:33281/ using document root <none> and password file <none>
I20260925 02:34:17.444227 25783 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:17.444275 25783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:17.444450 25783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:34:17.445590 25783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "8a5a6c1c0ccf49f3a7a59126c1745985"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "ff66d5bca9bbc0f9ff32fc038141b884"
server_key_iv: "bd44e2ea208d052c757f2c38797caedc"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.445953 25783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "8a5a6c1c0ccf49f3a7a59126c1745985"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "ff66d5bca9bbc0f9ff32fc038141b884"
server_key_iv: "bd44e2ea208d052c757f2c38797caedc"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.447338 25783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:34:17.448199 25798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.448387 25783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:34:17.448467 25783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "8a5a6c1c0ccf49f3a7a59126c1745985"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "ff66d5bca9bbc0f9ff32fc038141b884"
server_key_iv: "bd44e2ea208d052c757f2c38797caedc"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.448565 25783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:17.467517 25783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:17.470114 25783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:17.470314 25783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:17.470551 25783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:17.470908 25783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:17.470945 25783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.470969 25783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:17.471017 25783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.477695 25783 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:37881
I20260925 02:34:17.477743 25911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:37881 every 8 connection(s)
I20260925 02:34:17.478092 25783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:34:17.480513 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 25783
I20260925 02:34:17.480608 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:34:17.480964 10292 external_mini_cluster.cc:1428] Setting key d54cff968391ead3d518d629ab6b92ae
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:34:17.489274 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.478832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:36769 (local address 127.10.13.62:39519)
0925 02:34:17.478924 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.478926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.479132 (+ 206us) server_negotiation.cc:407] Connection header received
0925 02:34:17.479679 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.479681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.479694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.479761 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:17.480850 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.481607 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.482711 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.483045 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.483130 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.485827 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:17.485829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:17.485830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:17.485839 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:17.487549 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.488075 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.488076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.488077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.488142 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.488412 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.488413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.488414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.488567 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:17.488646 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.489051 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.489138 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:34:17.489423 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.478810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:36769)
0925 02:34:17.478992 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:17.479038 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:17.479563 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:17.479843 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:17.479847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:17.480466 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:17.480747 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.480753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.481734 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.481736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.482618 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.482621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.483165 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.483166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.483346 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.483887 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:17.483898 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:17.485723 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:17.487673 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:17.487681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:17.487684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:17.487988 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:17.488216 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:17.488216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:17.488217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:17.488337 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:17.488639 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:17.488644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:17.488782 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 02:34:17.489021 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 02:34:17.489179 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260925 02:34:17.490056 25912 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39519
I20260925 02:34:17.490170 25912 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
I20260925 02:34:17.490410 25912 heartbeater.cc:511] Master 127.10.13.62:39519 requested a full tablet report, sending...
I20260925 02:34:17.491005 25721 ts_manager.cc:194] Registered new tserver with Master: 8a5a6c1c0ccf49f3a7a59126c1745985 (127.10.13.1:37881)
I20260925 02:34:17.491655 25721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:36769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:34:17.508814 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39519
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:17.592476 25917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:17.592700 25917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:17.592730 25917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:17.594007 25917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:17.594055 25917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:17.594084 25917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:34:17.595561 25917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:17.595810 25917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:17.596076 25917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:17.601377 25923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:17.601428 25917 server_base.cc:1034] running on GCE node
W20260925 02:34:17.601500 25922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:17.601387 25925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:17.601809 25917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:17.602057 25917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:17.603222 25917 hybrid_clock.cc:648] HybridClock initialized: now 1790303657603210 us; error 29 us; skew 500 ppm
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:17.605808 25917 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:34:17.606280 25917 webserver.cc:460] Webserver started at http://127.10.13.2:37561/ using document root <none> and password file <none>
I20260925 02:34:17.606514 25917 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:17.606561 25917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:17.606693 25917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:34:17.607752 25917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "8e6e93f89c754a66a9a375ed9d062694"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "3ae6389f1628fe929157dd5b77481ee9"
server_key_iv: "101727cea129e3e2c49b3d345e1f1cd4"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.608117 25917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "8e6e93f89c754a66a9a375ed9d062694"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "3ae6389f1628fe929157dd5b77481ee9"
server_key_iv: "101727cea129e3e2c49b3d345e1f1cd4"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.609557 25917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:34:17.610483 25932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.610766 25917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:17.610888 25917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "8e6e93f89c754a66a9a375ed9d062694"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "3ae6389f1628fe929157dd5b77481ee9"
server_key_iv: "101727cea129e3e2c49b3d345e1f1cd4"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.610961 25917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:17.625851 25917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:17.628196 25917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:17.628348 25917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:17.628610 25917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:17.629001 25917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:17.629040 25917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.629065 25917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:17.629110 25917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.635341 25917 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:38377
I20260925 02:34:17.635386 26045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:38377 every 8 connection(s)
I20260925 02:34:17.635706 25917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:34:17.644150 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 25917
I20260925 02:34:17.644286 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:34:17.644542 10292 external_mini_cluster.cc:1428] Setting key 10cc12b53c02d4b8bb7df7715d6234c3
I20260925 02:34:17.645668 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.636485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:36101 (local address 127.10.13.62:39519)
0925 02:34:17.636551 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.636553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.636755 (+ 202us) server_negotiation.cc:407] Connection header received
0925 02:34:17.637375 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.637377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.637390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.637447 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:17.638425 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.638939 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.639942 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.640158 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.640240 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.642545 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:17.642547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:17.642547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:17.642555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:17.643907 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.644436 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.644438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.644438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.644501 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.644793 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.644794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.644794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.644995 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:17.645067 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.645432 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.645537 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:34:17.645762 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.636477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:36101)
0925 02:34:17.636629 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:17.636668 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:17.637262 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:17.637521 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:17.637524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:17.638109 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:17.638307 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.638312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.639037 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.639038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.639845 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.639848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.640248 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.640249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.640385 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.640981 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:17.640990 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:17.642431 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:17.644027 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:17.644035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:17.644038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:17.644360 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:17.644583 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:17.644584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:17.644584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:17.644717 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:17.645077 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:17.645085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:17.645274 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 02:34:17.645398 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 02:34:17.645558 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260925 02:34:17.646340 26046 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39519
I20260925 02:34:17.646463 26046 heartbeater.cc:464] Registering TS with master...
I20260925 02:34:17.646663 26046 heartbeater.cc:511] Master 127.10.13.62:39519 requested a full tablet report, sending...
I20260925 02:34:17.647158 25721 ts_manager.cc:194] Registered new tserver with Master: 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377)
I20260925 02:34:17.647567 25721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:36101
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:34:17.672996 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39519
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:17.756230 26051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:17.756474 26051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:17.756515 26051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:17.757683 26051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:17.757747 26051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:17.757792 26051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:34:17.759250 26051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:17.759536 26051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:17.759769 26051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:17.765129 26056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:17.765172 26051 server_base.cc:1034] running on GCE node
W20260925 02:34:17.765129 26059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:17.765115 26057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:17.765623 26051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:17.765897 26051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:17.767084 26051 hybrid_clock.cc:648] HybridClock initialized: now 1790303657767062 us; error 38 us; skew 500 ppm
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:17.769747 26051 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:34:17.770205 26051 webserver.cc:460] Webserver started at http://127.10.13.3:37393/ using document root <none> and password file <none>
I20260925 02:34:17.770466 26051 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:17.770540 26051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:17.770694 26051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:34:17.771785 26051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "16ca5b0694ac4269ab124c10564c88e6"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "a6a9b2fecd8f2a61918d74d23012226d"
server_key_iv: "39e04a083137b0ced24bce53bbacbfd9"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.772186 26051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "16ca5b0694ac4269ab124c10564c88e6"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "a6a9b2fecd8f2a61918d74d23012226d"
server_key_iv: "39e04a083137b0ced24bce53bbacbfd9"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.773693 26051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:34:17.774678 26066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.774861 26051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:17.774945 26051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "16ca5b0694ac4269ab124c10564c88e6"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "a6a9b2fecd8f2a61918d74d23012226d"
server_key_iv: "39e04a083137b0ced24bce53bbacbfd9"
server_key_version: "encryptionkey@0"
I20260925 02:34:17.775023 26051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:17.785670 26051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:17.788021 26051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:17.788137 26051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:17.788380 26051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:17.788708 26051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:17.788762 26051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.788805 26051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:17.788833 26051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:17.794724 26051 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:38469
I20260925 02:34:17.794801 26179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:38469 every 8 connection(s)
I20260925 02:34:17.795148 26051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:34:17.798528 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 26051
I20260925 02:34:17.798635 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:34:17.798828 10292 external_mini_cluster.cc:1428] Setting key 8c8398d4e7a5004bbba75ef81a380847
Sep 25 02:34:17 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303657, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:34:17.804625 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.795963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:56053 (local address 127.10.13.62:39519)
0925 02:34:17.796039 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.796041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.796255 (+ 214us) server_negotiation.cc:407] Connection header received
0925 02:34:17.796762 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.796763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.796776 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.796842 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:17.797904 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.798480 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.799665 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.799905 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.799957 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.802147 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:17.802148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:17.802149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:17.802156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:17.803432 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.803803 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.803804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.803805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.803856 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.804090 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.804090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.804090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.804199 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:17.804250 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.804495 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.804557 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:34:17.804800 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.795937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:56053)
0925 02:34:17.796122 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:17.796162 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:17.796634 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:17.797001 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:17.797005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:17.797572 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:17.797776 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.797781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.798577 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.798578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.799535 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.799538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.799975 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.799975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.800085 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.800548 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:17.800556 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:17.802054 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:17.803510 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:17.803518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:17.803519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:17.803738 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:17.803918 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:17.803919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:17.803919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:17.804032 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:17.804255 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:17.804260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:17.804384 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 02:34:17.804465 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:34:17.804588 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260925 02:34:17.805351 26180 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39519
I20260925 02:34:17.805473 26180 heartbeater.cc:464] Registering TS with master...
I20260925 02:34:17.805660 26180 heartbeater.cc:511] Master 127.10.13.62:39519 requested a full tablet report, sending...
I20260925 02:34:17.806061 25721 ts_manager.cc:194] Registered new tserver with Master: 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469)
I20260925 02:34:17.806403 25721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:56053
I20260925 02:34:17.810527 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:34:17.818413 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.811917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.10.13.62:39519)
0925 02:34:17.811978 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.811980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.812231 (+ 251us) server_negotiation.cc:407] Connection header received
0925 02:34:17.812324 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.812325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.812336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.812407 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:17.813012 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.813541 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.814515 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.814707 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.814792 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.815982 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:17.815984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:17.815984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:17.815991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:17.817207 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.817585 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.817586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.817587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.817647 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:17.817857 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:17.817858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:17.817859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:17.818020 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:17.818130 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.818234 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.818310 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:34:17.820282 25721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:34:17.822012 25721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:34:17.828938 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.824573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.10.13.1:37881)
0925 02:34:17.824860 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.824864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.825095 (+ 231us) server_negotiation.cc:407] Connection header received
0925 02:34:17.825197 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.825200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.825217 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.825364 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:17.825600 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.826712 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.827747 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.828472 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.828554 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.828621 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.828766 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.828807 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260925 02:34:17.828930 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.824685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:38377 (local address 127.0.0.1:33324)
0925 02:34:17.824918 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:17.824928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:17.824985 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:17.825214 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:17.825216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:17.825219 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:17.825295 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.825297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.826307 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.826308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.827702 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.827704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.828671 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.828672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.828745 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.828748 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:34:17.828792 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 02:34:17.828824 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":28,"thread_start_us":73,"threads_started":1}
I20260925 02:34:17.828930 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.824461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37881 (local address 127.0.0.1:36232)
0925 02:34:17.825021 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:17.825031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:17.825122 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:17.825468 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:17.825468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:17.825471 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:17.825512 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.825513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.826820 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.826821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.827650 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.827653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.828563 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.828564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.828648 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.828663 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:34:17.828738 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:34:17.828803 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":2,"thread_start_us":49,"threads_started":1}
I20260925 02:34:17.829053 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.824587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.10.13.3:38469)
0925 02:34:17.824834 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.824840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.824959 (+ 119us) server_negotiation.cc:407] Connection header received
0925 02:34:17.825118 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.825120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.825135 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.825211 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:17.825451 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.826569 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.827878 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.828564 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.828623 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.828669 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.828915 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.828967 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260925 02:34:17.829288 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.824826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.10.13.2:38377)
0925 02:34:17.825047 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.825050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.825060 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:34:17.825113 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.825115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.825133 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.825211 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:17.825383 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.826163 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.827804 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.828488 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.828578 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.828640 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.829173 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.829217 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":120,"threads_started":1}
I20260925 02:34:17.829335 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.824485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:38469 (local address 127.0.0.1:35894)
0925 02:34:17.824819 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:17.824898 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:17.824979 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:17.825297 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:17.825298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:17.825301 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:17.825363 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.825365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.826675 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.826676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.827799 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.827801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.828730 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.828731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.828797 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.828801 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:34:17.829166 (+ 365us) client_negotiation.cc:240] Negotiation successful
0925 02:34:17.829228 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":34,"thread_start_us":91,"threads_started":1}
I20260925 02:34:17.830049 25980 tablet_service.cc:1467] Processing CreateTablet for tablet 83130cc3f5254db2b55db97799c28cb5 (DEFAULT_TABLE table=test-table [id=747d3deda9274d87ad6957bbe7f2c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:17.830060 26114 tablet_service.cc:1467] Processing CreateTablet for tablet 83130cc3f5254db2b55db97799c28cb5 (DEFAULT_TABLE table=test-table [id=747d3deda9274d87ad6957bbe7f2c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:17.830267 25845 tablet_service.cc:1467] Processing CreateTablet for tablet 83130cc3f5254db2b55db97799c28cb5 (DEFAULT_TABLE table=test-table [id=747d3deda9274d87ad6957bbe7f2c22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:17.830448 26114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83130cc3f5254db2b55db97799c28cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:17.830448 25980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83130cc3f5254db2b55db97799c28cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:17.830561 25845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83130cc3f5254db2b55db97799c28cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:17.833477 26197 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Bootstrap starting.
I20260925 02:34:17.833846 26199 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: Bootstrap starting.
I20260925 02:34:17.834478 26197 tablet_bootstrap.cc:654] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:17.834592 26199 tablet_bootstrap.cc:654] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:17.835176 26197 log.cc:824] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:17.835203 26199 log.cc:824] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:17.835541 26198 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Bootstrap starting.
I20260925 02:34:17.836091 26199 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: No bootstrap required, opened a new log
I20260925 02:34:17.836170 26199 ts_tablet_manager.cc:1397] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:34:17.836261 26198 tablet_bootstrap.cc:654] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:17.836796 26198 log.cc:824] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:17.837522 26198 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: No bootstrap required, opened a new log
I20260925 02:34:17.837579 26198 ts_tablet_manager.cc:1397] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:34:17.837774 26199 raft_consensus.cc:348] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:17.837951 26199 raft_consensus.cc:374] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:17.837999 26199 raft_consensus.cc:729] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ca5b0694ac4269ab124c10564c88e6, State: Initialized, Role: FOLLOWER
I20260925 02:34:17.838124 26199 consensus_queue.cc:260] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:17.838434 26199 ts_tablet_manager.cc:1428] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:34:17.838486 26180 heartbeater.cc:503] Master 127.10.13.62:39519 was elected leader, sending a full tablet report...
I20260925 02:34:17.839084 26197 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: No bootstrap required, opened a new log
I20260925 02:34:17.839159 26197 ts_tablet_manager.cc:1397] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 02:34:17.839476 26198 raft_consensus.cc:348] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:17.839649 26198 raft_consensus.cc:374] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:17.839689 26198 raft_consensus.cc:729] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5a6c1c0ccf49f3a7a59126c1745985, State: Initialized, Role: FOLLOWER
I20260925 02:34:17.839820 26198 consensus_queue.cc:260] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:17.840126 26198 ts_tablet_manager.cc:1428] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:34:17.840173 25912 heartbeater.cc:503] Master 127.10.13.62:39519 was elected leader, sending a full tablet report...
I20260925 02:34:17.841346 26197 raft_consensus.cc:348] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:17.841513 26197 raft_consensus.cc:374] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:17.841538 26197 raft_consensus.cc:729] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6e93f89c754a66a9a375ed9d062694, State: Initialized, Role: FOLLOWER
I20260925 02:34:17.841661 26197 consensus_queue.cc:260] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:17.841892 26197 ts_tablet_manager.cc:1428] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 02:34:17.841972 26046 heartbeater.cc:503] Master 127.10.13.62:39519 was elected leader, sending a full tablet report...
W20260925 02:34:17.886487 26047 tablet.cc:2369] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:34:17.978847 25913 tablet.cc:2369] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:34:17.992316 26205 raft_consensus.cc:482] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:34:17.992504 26205 raft_consensus.cc:504] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:17.992971 26205 leader_election.cc:288] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a5a6c1c0ccf49f3a7a59126c1745985 (127.10.13.1:37881), 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469)
I20260925 02:34:17.995985 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.993166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37881 (local address 127.10.13.2:41717)
0925 02:34:17.993399 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:17.993410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:17.993470 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:17.993645 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:17.993646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:17.993648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:17.993754 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.993756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.994404 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.994405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.995139 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.995141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.995704 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.995706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.995786 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.995790 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:34:17.995832 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 02:34:17.995871 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":60,"thread_start_us":115,"threads_started":1}
I20260925 02:34:17.995992 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.993244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:41717 (local address 127.10.13.1:37881)
0925 02:34:17.993301 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.993303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.993478 (+ 175us) server_negotiation.cc:407] Connection header received
0925 02:34:17.993560 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.993561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.993574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.993635 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:17.993845 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.994295 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.995268 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.995621 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.995676 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.995687 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.995862 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.995922 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:34:17.996060 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.993243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:54441 (local address 127.10.13.3:38469)
0925 02:34:17.993294 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:17.993296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:17.993310 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:34:17.993392 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:17.993393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:17.993406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:17.993474 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:17.993727 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.994274 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.995232 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.995636 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.995727 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.995737 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:17.995934 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 02:34:17.995973 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:34:17.996086 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:17.993164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:38469 (local address 127.10.13.2:54441)
0925 02:34:17.993241 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:17.993253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:17.993307 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:17.993540 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:17.993542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:17.993544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:17.993629 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.993632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.994371 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.994372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.995130 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:17.995132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:17.995757 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:17.995758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:17.995846 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:17.995850 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:34:17.995912 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:34:17.995953 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 02:34:17.996634 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83130cc3f5254db2b55db97799c28cb5" candidate_uuid: "8e6e93f89c754a66a9a375ed9d062694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ca5b0694ac4269ab124c10564c88e6" is_pre_election: true
I20260925 02:34:17.996634 25866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83130cc3f5254db2b55db97799c28cb5" candidate_uuid: "8e6e93f89c754a66a9a375ed9d062694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" is_pre_election: true
I20260925 02:34:17.996803 26134 raft_consensus.cc:2454] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e6e93f89c754a66a9a375ed9d062694 in term 0.
I20260925 02:34:17.996803 25866 raft_consensus.cc:2454] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e6e93f89c754a66a9a375ed9d062694 in term 0.
I20260925 02:34:17.997099 25936 leader_election.cc:302] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5a6c1c0ccf49f3a7a59126c1745985, 8e6e93f89c754a66a9a375ed9d062694; no voters:
I20260925 02:34:17.997295 26205 raft_consensus.cc:2781] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:34:17.997346 26205 raft_consensus.cc:482] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:34:17.997364 26205 raft_consensus.cc:3037] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:17.998111 26205 raft_consensus.cc:504] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:17.998274 26205 leader_election.cc:288] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [CANDIDATE]: Term 1 election: Requested vote from peers 8a5a6c1c0ccf49f3a7a59126c1745985 (127.10.13.1:37881), 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469)
I20260925 02:34:17.998430 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83130cc3f5254db2b55db97799c28cb5" candidate_uuid: "8e6e93f89c754a66a9a375ed9d062694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ca5b0694ac4269ab124c10564c88e6"
I20260925 02:34:17.998451 25866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83130cc3f5254db2b55db97799c28cb5" candidate_uuid: "8e6e93f89c754a66a9a375ed9d062694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985"
I20260925 02:34:17.998515 26134 raft_consensus.cc:3037] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:17.998522 25866 raft_consensus.cc:3037] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:17.999405 26134 raft_consensus.cc:2454] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e6e93f89c754a66a9a375ed9d062694 in term 1.
I20260925 02:34:17.999397 25866 raft_consensus.cc:2454] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e6e93f89c754a66a9a375ed9d062694 in term 1.
I20260925 02:34:17.999629 25934 leader_election.cc:302] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16ca5b0694ac4269ab124c10564c88e6, 8a5a6c1c0ccf49f3a7a59126c1745985, 8e6e93f89c754a66a9a375ed9d062694; no voters:
I20260925 02:34:17.999738 26205 raft_consensus.cc:2781] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:34:17.999938 26205 raft_consensus.cc:686] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 LEADER]: Becoming Leader. State: Replica: 8e6e93f89c754a66a9a375ed9d062694, State: Running, Role: LEADER
I20260925 02:34:18.000057 26205 consensus_queue.cc:237] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:18.001049 25720 catalog_manager.cc:5495] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2). New cstate: current_term: 1 leader_uuid: "8e6e93f89c754a66a9a375ed9d062694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } health_report { overall_health: HEALTHY } } }
I20260925 02:34:18.014292 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:18.011563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.10.13.2:38377)
0925 02:34:18.011630 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:18.011632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:18.011641 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:34:18.011682 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:18.011683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:18.011702 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:18.011774 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:34:18.011990 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:18.012535 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:18.013475 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:18.013658 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:18.013714 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:18.013973 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:34:18.014070 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:34:18.014134 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:18.014216 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:34:18.014233 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:34:18.016181 25866 raft_consensus.cc:1261] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Refusing update from remote peer 8e6e93f89c754a66a9a375ed9d062694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:34:18.016191 26134 raft_consensus.cc:1261] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 1 FOLLOWER]: Refusing update from remote peer 8e6e93f89c754a66a9a375ed9d062694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:34:18.016692 26207 consensus_queue.cc:1035] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:34:18.016934 26205 consensus_queue.cc:1035] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:34:18.020164 26212 mvcc.cc:205] Tried to move back new op lower bound from 7333083783231422464 to 7333083783169003520. Current Snapshot: MvccSnapshot[applied={T|T < 7333083783231422464}]
I20260925 02:34:18.020190 26211 mvcc.cc:205] Tried to move back new op lower bound from 7333083783231422464 to 7333083783169003520. Current Snapshot: MvccSnapshot[applied={T|T < 7333083783231422464 or (T in {7333083783231422464})}]
I20260925 02:34:18.021633 25720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:34:18.021744 25720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:34:18.022529 25720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Sending DeleteTablet for 3 replicas of tablet 83130cc3f5254db2b55db97799c28cb5
I20260925 02:34:18.022852 25845 tablet_service.cc:1514] Processing DeleteTablet for tablet 83130cc3f5254db2b55db97799c28cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:18 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:36232
I20260925 02:34:18.022976 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 25783
I20260925 02:34:18.022941 26114 tablet_service.cc:1514] Processing DeleteTablet for tablet 83130cc3f5254db2b55db97799c28cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:18 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:35894
I20260925 02:34:18.022966 25980 tablet_service.cc:1514] Processing DeleteTablet for tablet 83130cc3f5254db2b55db97799c28cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:18 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:33324
I20260925 02:34:18.023833 26221 tablet_replica.cc:330] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: stopping tablet replica
I20260925 02:34:18.023986 26221 raft_consensus.cc:2229] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:34:18.024142 26221 raft_consensus.cc:2258] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:34:18.024309 26222 tablet_replica.cc:330] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: stopping tablet replica
I20260925 02:34:18.024408 26222 raft_consensus.cc:2229] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:34:18.024534 26222 raft_consensus.cc:2258] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:34:18.025034 26221 ts_tablet_manager.cc:1905] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:34:18.027249 26221 ts_tablet_manager.cc:1918] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:34:18.027334 26221 log.cc:1196] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/83130cc3f5254db2b55db97799c28cb5
I20260925 02:34:18.027611 26221 ts_tablet_manager.cc:1939] T 83130cc3f5254db2b55db97799c28cb5 P 8e6e93f89c754a66a9a375ed9d062694: Deleting consensus metadata
I20260925 02:34:18.028116 25706 catalog_manager.cc:4841] TS 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377): tablet 83130cc3f5254db2b55db97799c28cb5 (table test-table [id=747d3deda9274d87ad6957bbe7f2c22d]) successfully deleted
I20260925 02:34:18.028170 26222 ts_tablet_manager.cc:1905] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:34:18.029801 25708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:37881 (error 108)
I20260925 02:34:18.030071 26222 ts_tablet_manager.cc:1918] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:34:18.030222 26222 log.cc:1196] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/83130cc3f5254db2b55db97799c28cb5
I20260925 02:34:18.030331 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:18.030212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37881 (local address 127.0.0.1:36240)
0925 02:34:18.030255 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:18.030295 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 02:34:18.030413 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 25917
W20260925 02:34:18.030520 25708 catalog_manager.cc:4584] TS 8a5a6c1c0ccf49f3a7a59126c1745985 (127.10.13.1:37881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83130cc3f5254db2b55db97799c28cb5: Network error: Client connection negotiation failed: client connection to 127.10.13.1:37881: connect: Connection refused (error 111)
I20260925 02:34:18.031070 26222 ts_tablet_manager.cc:1939] T 83130cc3f5254db2b55db97799c28cb5 P 16ca5b0694ac4269ab124c10564c88e6: Deleting consensus metadata
I20260925 02:34:18.031502 25706 catalog_manager.cc:4841] TS 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469): tablet 83130cc3f5254db2b55db97799c28cb5 (table test-table [id=747d3deda9274d87ad6957bbe7f2c22d]) successfully deleted
I20260925 02:34:18.036432 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 26051
I20260925 02:34:18.042364 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 25689
I20260925 02:34:38.048190 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39519
--webserver_interface=127.10.13.62
--webserver_port=44385
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:38.138913 26224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:38.139125 26224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:38.139148 26224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:38.140435 26224 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:34:38.140477 26224 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:34:38.140491 26224 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:34:38.140502 26224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:34:38.140515 26224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:38.140528 26224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:38.140578 26224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:34:38.140601 26224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:34:38.142175 26224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:38.142424 26224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:38.142688 26224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:34:38.151894 26224 server_base.cc:1034] running on GCE node
W20260925 02:34:38.151852 26233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:38.151835 26231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:38.151906 26230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:38.152369 26224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:38.152755 26224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:38.153927 26224 hybrid_clock.cc:648] HybridClock initialized: now 1790303678153917 us; error 30 us; skew 500 ppm
Sep 25 02:34:38 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303678, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:38.166637 26224 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:34:38.167311 26224 webserver.cc:460] Webserver started at http://127.10.13.62:44385/ using document root <none> and password file <none>
I20260925 02:34:38.167625 26224 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:38.167728 26224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:38.169425 26224 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:34:38.170305 26240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.170549 26224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:38.170684 26224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "e0ed57e3d44c453a9778129e3005c039"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "60f04478e590c5ec6a9de6066a740c1e"
server_key_iv: "93bbb3e765c3b6e35e7a206a8dd41431"
server_key_version: "encryptionkey@0"
I20260925 02:34:38.171168 26224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:38.181049 26224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:38.183525 26224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:38.183707 26224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:38.188207 26224 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:39519
I20260925 02:34:38.188295 26292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:39519 every 8 connection(s)
I20260925 02:34:38.188742 26224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:34:38.190471 26293 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:34:38.191124 26293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Bootstrap starting.
I20260925 02:34:38.194273 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 26224
I20260925 02:34:38.194610 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:37881
--local_ip_for_outbound_sockets=127.10.13.1
--tserver_master_addrs=127.10.13.62:39519
--webserver_port=33281
--webserver_interface=127.10.13.1
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:34:38.200830 26293 log.cc:824] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:38.202992 26293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:34:38.203356 26293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Bootstrap complete.
I20260925 02:34:38.205873 26293 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:38.206250 26293 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ed57e3d44c453a9778129e3005c039, State: Initialized, Role: FOLLOWER
I20260925 02:34:38.206532 26293 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:38.206646 26293 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:34:38.206712 26293 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:34:38.206768 26293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:34:38.207880 26293 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:38.208016 26293 leader_election.cc:302] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ed57e3d44c453a9778129e3005c039; no voters:
I20260925 02:34:38.208236 26293 leader_election.cc:288] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:34:38.208279 26297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:34:38.208401 26297 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 2 LEADER]: Becoming Leader. State: Replica: e0ed57e3d44c453a9778129e3005c039, State: Running, Role: LEADER
I20260925 02:34:38.208565 26293 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:34:38.208556 26297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:38.209003 26299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ed57e3d44c453a9778129e3005c039. Latest consensus state: current_term: 2 leader_uuid: "e0ed57e3d44c453a9778129e3005c039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } } }
I20260925 02:34:38.209169 26298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0ed57e3d44c453a9778129e3005c039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } } }
I20260925 02:34:38.209235 26299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: This master's current role is: LEADER
I20260925 02:34:38.209335 26298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: This master's current role is: LEADER
I20260925 02:34:38.209590 26302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:34:38.210529 26302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:34:38.210690 26302 catalog_manager.cc:1259] Loaded cluster ID: 0dd16d79f7f346669130e09ad68b9354
I20260925 02:34:38.210812 26302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:34:38.212359 26302 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:34:38.212620 26302 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Loaded TSK: 0
I20260925 02:34:38.229252 26302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Generated new TSK 1
I20260925 02:34:38.229389 26302 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:34:38.283895 26295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:38.284142 26295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:38.284185 26295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:38.285458 26295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:38.285522 26295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:38.285568 26295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:34:38.287076 26295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:37881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=33281
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:38.287345 26295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:38.287582 26295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:38.293799 26321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:38.293799 26320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:38.293794 26323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:38.294386 26295 server_base.cc:1034] running on GCE node
I20260925 02:34:38.294596 26295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:38.294860 26295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:38.296051 26295 hybrid_clock.cc:648] HybridClock initialized: now 1790303678296036 us; error 31 us; skew 500 ppm
Sep 25 02:34:38 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303678, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:38.298734 26295 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:34:38.299278 26295 webserver.cc:460] Webserver started at http://127.10.13.1:33281/ using document root <none> and password file <none>
I20260925 02:34:38.299561 26295 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:38.299641 26295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:38.301177 26295 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:38.301946 26330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.302151 26295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:38.302238 26295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "8a5a6c1c0ccf49f3a7a59126c1745985"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "ff66d5bca9bbc0f9ff32fc038141b884"
server_key_iv: "bd44e2ea208d052c757f2c38797caedc"
server_key_version: "encryptionkey@0"
I20260925 02:34:38.302626 26295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:38.321880 26295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:38.324338 26295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:38.324520 26295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:38.324787 26295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:38.325315 26337 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:34:38.327389 26295 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:34:38.327474 26295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:34:38.327546 26295 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:34:38.328286 26295 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:34:38.328346 26295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:34:38.328434 26337 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Bootstrap starting.
I20260925 02:34:38.335888 26295 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:37881
I20260925 02:34:38.335927 26444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:37881 every 8 connection(s)
I20260925 02:34:38.336407 26295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:34:38.340134 26337 log.cc:824] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:38.340806 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 26295
I20260925 02:34:38.341125 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:38377
--local_ip_for_outbound_sockets=127.10.13.2
--tserver_master_addrs=127.10.13.62:39519
--webserver_port=37561
--webserver_interface=127.10.13.2
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:34:38.344265 26337 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:34:38.344679 26337 tablet_bootstrap.cc:492] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Bootstrap complete.
I20260925 02:34:38.346048 26337 ts_tablet_manager.cc:1397] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260925 02:34:38.347775 26337 raft_consensus.cc:348] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:38.348201 26337 raft_consensus.cc:729] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5a6c1c0ccf49f3a7a59126c1745985, State: Initialized, Role: FOLLOWER
I20260925 02:34:38.348387 26337 consensus_queue.cc:260] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } } peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } }
I20260925 02:34:38.348748 26337 ts_tablet_manager.cc:1428] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
Sep 25 02:34:38 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303678, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:34:38.365015 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.337209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:57425)
0925 02:34:38.340317 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.340383 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:38.342067 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:38.342723 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:38.342727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:38.352420 (+ 9693us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:38.352908 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.352918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.355510 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.355511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.356819 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.356823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.357792 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.357793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.357938 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.358661 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:38.358674 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:38.360738 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:38.363111 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:38.363120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:38.363123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:38.363440 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:38.363712 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:38.363714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:38.363714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:38.363863 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:38.364252 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:38.364257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:38.364421 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 02:34:38.364524 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:34:38.364717 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3044,"thread_start_us":90,"threads_started":1}
I20260925 02:34:38.365027 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.337282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:57425 (local address 127.10.13.62:39519)
0925 02:34:38.340744 (+ 3462us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.340755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.340810 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:34:38.342376 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.342382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.342458 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.342582 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:38.353175 (+ 10593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.355364 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.356975 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.357678 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.357814 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.360944 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:38.360950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:38.360950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:38.360964 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:38.363019 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.363543 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.363547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.363547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.363644 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.363945 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.363946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.363947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.364184 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:38.364251 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.364665 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.364797 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3323,"thread_start_us":104,"threads_started":1}
I20260925 02:34:38.365860 26445 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39519
I20260925 02:34:38.365968 26445 heartbeater.cc:464] Registering TS with master...
I20260925 02:34:38.366159 26445 heartbeater.cc:511] Master 127.10.13.62:39519 requested a full tablet report, sending...
I20260925 02:34:38.366988 26257 ts_manager.cc:194] Registered new tserver with Master: 8a5a6c1c0ccf49f3a7a59126c1745985 (127.10.13.1:37881)
I20260925 02:34:38.368264 26257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:57425
I20260925 02:34:38.369475 26445 heartbeater.cc:503] Master 127.10.13.62:39519 was elected leader, sending a full tablet report...
Sep 25 02:34:38 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303678, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.1@KRBTEST.COM
I20260925 02:34:38.376664 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.367748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.10.13.1:37881)
0925 02:34:38.368110 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.368114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.368122 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:34:38.368148 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.368150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.368162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.368242 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:38.368810 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.369695 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.370539 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.371081 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.371160 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.373465 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:38.373472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:38.373473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:38.373480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:38.375349 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.375868 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.375871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.375871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.375929 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.376130 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.376130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.376131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.376269 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:38.376332 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.376519 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.376596 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":61,"threads_started":1}
I20260925 02:34:38.376708 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.367692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37881 (local address 127.0.0.1:56476)
0925 02:34:38.367929 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.367955 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:38.368008 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:38.368248 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:38.368254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:38.368648 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:38.368711 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.368714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.369782 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.369788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.370460 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.370463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.371169 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.371169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.371230 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.371771 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:38.371777 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:38.373358 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:38.375435 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:38.375442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:38.375444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:38.375738 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:38.375997 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:38.375998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:38.375998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:38.376067 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:38.376333 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:38.376340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:38.376397 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 02:34:38.376488 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:34:38.376612 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260925 02:34:38.377601 26366 tablet_service.cc:1514] Processing DeleteTablet for tablet 83130cc3f5254db2b55db97799c28cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:18 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:56476
I20260925 02:34:38.378314 26454 tablet_replica.cc:330] stopping tablet replica
I20260925 02:34:38.378439 26454 raft_consensus.cc:2229] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:34:38.378523 26454 raft_consensus.cc:2258] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:34:38.378912 26454 ts_tablet_manager.cc:1905] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:34:38.381170 26454 ts_tablet_manager.cc:1918] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:34:38.381268 26454 log.cc:1196] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal/wals/83130cc3f5254db2b55db97799c28cb5
I20260925 02:34:38.381552 26454 ts_tablet_manager.cc:1939] T 83130cc3f5254db2b55db97799c28cb5 P 8a5a6c1c0ccf49f3a7a59126c1745985: Deleting consensus metadata
I20260925 02:34:38.382134 26244 catalog_manager.cc:4841] TS 8a5a6c1c0ccf49f3a7a59126c1745985 (127.10.13.1:37881): tablet 83130cc3f5254db2b55db97799c28cb5 (table test-table [id=747d3deda9274d87ad6957bbe7f2c22d]) successfully deleted
W20260925 02:34:38.442390 26450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:38.442615 26450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:38.442639 26450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:38.443845 26450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:38.443892 26450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:38.443918 26450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:34:38.445446 26450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:38377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=37561
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:38.445716 26450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:38.446017 26450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:38.451325 26459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:38.451443 26450 server_base.cc:1034] running on GCE node
W20260925 02:34:38.451426 26460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:38.451325 26462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:38.451845 26450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:38.452114 26450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:38.453316 26450 hybrid_clock.cc:648] HybridClock initialized: now 1790303678453300 us; error 39 us; skew 500 ppm
Sep 25 02:34:38 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303678, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:38.456002 26450 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:34:38.456466 26450 webserver.cc:460] Webserver started at http://127.10.13.2:37561/ using document root <none> and password file <none>
I20260925 02:34:38.456708 26450 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:38.456787 26450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:38.458433 26450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:34:38.459112 26469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.459312 26450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:34:38.459431 26450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "8e6e93f89c754a66a9a375ed9d062694"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "3ae6389f1628fe929157dd5b77481ee9"
server_key_iv: "101727cea129e3e2c49b3d345e1f1cd4"
server_key_version: "encryptionkey@0"
I20260925 02:34:38.459805 26450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:38.472988 26450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:38.475569 26450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:38.475752 26450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:38.476027 26450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:38.476372 26450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:38.476433 26450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.476478 26450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:38.476506 26450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.482669 26450 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:38377
I20260925 02:34:38.482774 26582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:38377 every 8 connection(s)
I20260925 02:34:38.483129 26450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:34:38.486708 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 26450
I20260925 02:34:38.487071 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:38469
--local_ip_for_outbound_sockets=127.10.13.3
--tserver_master_addrs=127.10.13.62:39519
--webserver_port=37393
--webserver_interface=127.10.13.3
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:34:38 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303678, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:34:38.495036 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.484038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:54607 (local address 127.10.13.62:39519)
0925 02:34:38.484118 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.484121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.484358 (+ 237us) server_negotiation.cc:407] Connection header received
0925 02:34:38.484951 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.484952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.484964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.485028 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:38.486040 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.486591 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.488232 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.488508 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.488601 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.491505 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:38.491506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:38.491507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:38.491518 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:38.493374 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.493875 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.493876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.493877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.493946 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.494241 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.494242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.494243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.494419 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:38.494498 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.494783 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.494894 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:34:38.495076 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.484038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:54607)
0925 02:34:38.484211 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.484257 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:38.484772 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:38.485103 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:38.485107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:38.485706 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:38.485918 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.485923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.486709 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.486711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.488120 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.488124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.488626 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.488628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.488766 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.489461 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:38.489470 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:38.491375 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:38.493486 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:38.493494 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:38.493497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:38.493784 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:38.494026 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:38.494027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:38.494027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:38.494166 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:38.494495 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:38.494500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:38.494638 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 02:34:38.494733 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:34:38.494880 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260925 02:34:38.495698 26583 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39519
I20260925 02:34:38.495826 26583 heartbeater.cc:464] Registering TS with master...
I20260925 02:34:38.496052 26583 heartbeater.cc:511] Master 127.10.13.62:39519 requested a full tablet report, sending...
I20260925 02:34:38.496551 26257 ts_manager.cc:194] Registered new tserver with Master: 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377)
I20260925 02:34:38.497067 26257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:54607
W20260925 02:34:38.574590 26586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:38.574800 26586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:38.574833 26586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:38.576030 26586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:38.576077 26586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:38.576105 26586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:34:38.577747 26586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:38469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=37393
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:38.578016 26586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:38.578265 26586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:38.583520 26592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:38.583592 26591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:38.583529 26594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:38.583833 26586 server_base.cc:1034] running on GCE node
I20260925 02:34:38.584052 26586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:38.584335 26586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:38.585507 26586 hybrid_clock.cc:648] HybridClock initialized: now 1790303678585501 us; error 38 us; skew 500 ppm
Sep 25 02:34:38 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303678, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:38.588174 26586 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:34:38.588645 26586 webserver.cc:460] Webserver started at http://127.10.13.3:37393/ using document root <none> and password file <none>
I20260925 02:34:38.588979 26586 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:38.589069 26586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:38.590459 26586 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:38.591078 26601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.591295 26586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:38.591439 26586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "16ca5b0694ac4269ab124c10564c88e6"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "a6a9b2fecd8f2a61918d74d23012226d"
server_key_iv: "39e04a083137b0ced24bce53bbacbfd9"
server_key_version: "encryptionkey@0"
I20260925 02:34:38.591794 26586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:38.613197 26586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:38.615269 26586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:38.615460 26586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:38.615713 26586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:38.616035 26586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:38.616091 26586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.616132 26586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:38.616163 26586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.622293 26586 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:38469
I20260925 02:34:38.622337 26714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:38469 every 8 connection(s)
I20260925 02:34:38.622737 26586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:34:38.623553 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 26586
Sep 25 02:34:38 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303678, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:34:38.631652 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.624030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.10.13.62:39519)
0925 02:34:38.624621 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.624624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.624632 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:34:38.624657 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.624658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.624669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.624756 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:38.625448 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.626085 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.626943 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.627247 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.627319 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.628680 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:38.628681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:38.628681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:38.628689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:38.630261 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.630704 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.630706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.630706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.630831 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.631049 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.631050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.631051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.631214 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:38.631414 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.631457 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.631541 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":89,"threads_started":1}
I20260925 02:34:38.633365 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.623505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:58939 (local address 127.10.13.62:39519)
0925 02:34:38.623557 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.623564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.623885 (+ 321us) server_negotiation.cc:407] Connection header received
0925 02:34:38.624541 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.624542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.624552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.624607 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:38.625801 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.626368 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.627584 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.627769 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.627839 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.630300 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:38.630302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:38.630303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:38.630310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:38.631904 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.632360 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.632361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.632362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.632424 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.632710 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.632711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.632712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.632819 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:38.632905 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.633194 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.633280 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:34:38.633584 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.623531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:58939)
0925 02:34:38.623730 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.623795 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:38.624434 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:38.624726 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:38.624731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:38.625437 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:34:38.625706 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.625713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.626465 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.626466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.627501 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.627504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.627863 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.627864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.628001 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.628521 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:34:38.628529 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:34:38.630186 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:34:38.631990 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:38.631997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:38.631999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:38.632276 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:38.632502 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:34:38.632502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:34:38.632503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:34:38.632647 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:34:38.632912 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:34:38.632920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:34:38.633058 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 02:34:38.633172 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:34:38.633333 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260925 02:34:38.634167 26715 heartbeater.cc:347] Connected to a master server at 127.10.13.62:39519
I20260925 02:34:38.634287 26715 heartbeater.cc:464] Registering TS with master...
I20260925 02:34:38.634479 26715 heartbeater.cc:511] Master 127.10.13.62:39519 requested a full tablet report, sending...
I20260925 02:34:38.634889 26257 ts_manager.cc:194] Registered new tserver with Master: 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469)
I20260925 02:34:38.635264 26257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:58939
I20260925 02:34:38.643551 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:34:38.646373 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.643930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.10.13.62:39519)
0925 02:34:38.643984 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.643986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.644089 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:34:38.644141 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.644142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.644175 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.644239 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:34:38.644359 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.644811 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.645636 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.645846 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.645908 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.646114 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:34:38.646161 (+ 47us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 02:34:38.646305 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57580: token expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 02:34:38.646423 26448 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57580: token expired
W20260925 02:34:38.646713 26183 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.13.62:39519)
I20260925 02:34:38.652894 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.646961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.10.13.62:39519)
0925 02:34:38.647022 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.647024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.647033 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:34:38.647056 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.647057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.647066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.647151 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:34:38.647759 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.648199 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.648968 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.649154 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.649230 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.650387 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:34:38.650389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:34:38.650389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:34:38.650397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:34:38.651686 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.652110 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.652111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.652112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.652172 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:34:38.652379 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:34:38.652380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:34:38.652380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:34:38.652489 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:34:38.652594 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.652695 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.652776 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:34:38.654264 26257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:34:38.655974 26257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:34:38.658386 26366 tablet_service.cc:1467] Processing CreateTablet for tablet e8f1dbdf3e534160b07f9f4d5422d153 (DEFAULT_TABLE table=test-table [id=4031565db0964396a9aa47b7b4519c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:38.658607 26366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f1dbdf3e534160b07f9f4d5422d153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:38.660426 26337 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Bootstrap starting.
I20260925 02:34:38.660879 26337 tablet_bootstrap.cc:654] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:38.662040 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.657981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:38469 (local address 127.0.0.1:35290)
0925 02:34:38.658017 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.658029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:38.658064 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:38.658534 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:38.658535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:38.658537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:38.658580 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.658582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.659679 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.659680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.660608 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.660610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.661727 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.661728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.661837 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.661860 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:34:38.661924 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:34:38.661972 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 02:34:38.662045 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.658056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.10.13.3:38469)
0925 02:34:38.658345 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.658350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.658362 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:34:38.658402 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.658404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.658420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.658510 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:38.658647 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.659586 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.660761 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.661643 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.661717 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.661801 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.661949 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.661982 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":127,"threads_started":1}
I20260925 02:34:38.662542 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.658086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:38377 (local address 127.0.0.1:45244)
0925 02:34:38.658237 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.658247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:38.658310 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:38.658938 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:38.658939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:38.658941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:38.658983 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.658985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.660267 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.660268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.661312 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.661315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.662310 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.662311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.662396 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.662400 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:34:38.662465 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:34:38.662495 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260925 02:34:38.662599 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.658195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.10.13.2:38377)
0925 02:34:38.658682 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.658687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.658698 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:34:38.658737 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.658739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.658754 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.658833 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:38.659064 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.660173 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.661409 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.662229 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.662300 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.662353 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.662483 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.662522 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":92,"threads_started":1}
I20260925 02:34:38.662825 26649 tablet_service.cc:1467] Processing CreateTablet for tablet e8f1dbdf3e534160b07f9f4d5422d153 (DEFAULT_TABLE table=test-table [id=4031565db0964396a9aa47b7b4519c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:38.663223 26649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f1dbdf3e534160b07f9f4d5422d153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:38.663367 26517 tablet_service.cc:1467] Processing CreateTablet for tablet e8f1dbdf3e534160b07f9f4d5422d153 (DEFAULT_TABLE table=test-table [id=4031565db0964396a9aa47b7b4519c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:34:38.663765 26517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f1dbdf3e534160b07f9f4d5422d153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:34:38.666321 26725 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: Bootstrap starting.
I20260925 02:34:38.666433 26337 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: No bootstrap required, opened a new log
I20260925 02:34:38.666494 26337 ts_tablet_manager.cc:1397] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 02:34:38.666695 26337 raft_consensus.cc:348] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.666783 26337 raft_consensus.cc:374] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:38.666850 26337 raft_consensus.cc:729] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5a6c1c0ccf49f3a7a59126c1745985, State: Initialized, Role: FOLLOWER
I20260925 02:34:38.666851 26726 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Bootstrap starting.
I20260925 02:34:38.666916 26337 consensus_queue.cc:260] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.667021 26337 ts_tablet_manager.cc:1428] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:38.667126 26725 tablet_bootstrap.cc:654] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:38.667506 26726 tablet_bootstrap.cc:654] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Neither blocks nor log segments found. Creating new log.
I20260925 02:34:38.667850 26725 log.cc:824] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:38.668148 26726 log.cc:824] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:38.668689 26725 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: No bootstrap required, opened a new log
I20260925 02:34:38.668784 26725 ts_tablet_manager.cc:1397] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:34:38.669028 26726 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: No bootstrap required, opened a new log
I20260925 02:34:38.669082 26726 ts_tablet_manager.cc:1397] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:34:38.670614 26725 raft_consensus.cc:348] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.670670 26726 raft_consensus.cc:348] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.670776 26725 raft_consensus.cc:374] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:38.670799 26726 raft_consensus.cc:374] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:34:38.670828 26725 raft_consensus.cc:729] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ca5b0694ac4269ab124c10564c88e6, State: Initialized, Role: FOLLOWER
I20260925 02:34:38.670866 26726 raft_consensus.cc:729] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6e93f89c754a66a9a375ed9d062694, State: Initialized, Role: FOLLOWER
I20260925 02:34:38.670980 26726 consensus_queue.cc:260] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.670974 26725 consensus_queue.cc:260] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.671289 26726 ts_tablet_manager.cc:1428] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:34:38.671270 26715 heartbeater.cc:503] Master 127.10.13.62:39519 was elected leader, sending a full tablet report...
I20260925 02:34:38.671298 26725 ts_tablet_manager.cc:1428] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 02:34:38.671312 26583 heartbeater.cc:503] Master 127.10.13.62:39519 was elected leader, sending a full tablet report...
W20260925 02:34:38.733845 26584 tablet.cc:2369] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:34:38.754500 26451 raft_consensus.cc:482] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:34:38.754769 26451 raft_consensus.cc:504] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.755213 26451 leader_election.cc:288] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469), 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377)
I20260925 02:34:38.758487 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.755689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:34971 (local address 127.10.13.2:38377)
0925 02:34:38.755773 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.755775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.755846 (+ 71us) server_negotiation.cc:407] Connection header received
0925 02:34:38.755900 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.755901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.755914 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.755982 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:38.756182 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.756676 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.757688 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.758092 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.758151 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.758160 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.758361 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.758408 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:34:38.758522 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.755594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:54213 (local address 127.10.13.3:38469)
0925 02:34:38.755646 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.755648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.755660 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:34:38.755741 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.755742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.755755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.755811 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:34:38.756048 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.756578 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.757736 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.758112 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.758204 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.758215 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.758411 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.758456 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 02:34:38.758519 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.755479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:38469 (local address 127.10.13.1:54213)
0925 02:34:38.755551 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.755568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:38.755645 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:38.755879 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:38.755880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:38.755882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:38.755962 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.755965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.756657 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.756658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.757615 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.757618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.758217 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.758218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.758303 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.758306 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:34:38.758391 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:34:38.758424 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":65}
I20260925 02:34:38.758533 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.755613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:38377 (local address 127.10.13.1:34971)
0925 02:34:38.755791 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.755800 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:38.755849 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:38.755993 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:34:38.755994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:34:38.756014 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:34:38.756104 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.756105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.756756 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.756757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.757597 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:34:38.757600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.758200 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.758201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:34:38.758269 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.758287 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 02:34:38.758330 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 02:34:38.758387 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260925 02:34:38.758894 26537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f1dbdf3e534160b07f9f4d5422d153" candidate_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e6e93f89c754a66a9a375ed9d062694" is_pre_election: true
I20260925 02:34:38.758903 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f1dbdf3e534160b07f9f4d5422d153" candidate_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ca5b0694ac4269ab124c10564c88e6" is_pre_election: true
I20260925 02:34:38.759058 26537 raft_consensus.cc:2454] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5a6c1c0ccf49f3a7a59126c1745985 in term 0.
I20260925 02:34:38.759073 26669 raft_consensus.cc:2454] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5a6c1c0ccf49f3a7a59126c1745985 in term 0.
I20260925 02:34:38.759305 26332 leader_election.cc:302] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ca5b0694ac4269ab124c10564c88e6, 8a5a6c1c0ccf49f3a7a59126c1745985; no voters:
I20260925 02:34:38.759464 26451 raft_consensus.cc:2781] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:34:38.759536 26451 raft_consensus.cc:482] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:34:38.759553 26451 raft_consensus.cc:3037] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:38.760310 26451 raft_consensus.cc:504] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.760454 26451 leader_election.cc:288] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 1 election: Requested vote from peers 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469), 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377)
I20260925 02:34:38.760659 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f1dbdf3e534160b07f9f4d5422d153" candidate_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ca5b0694ac4269ab124c10564c88e6"
I20260925 02:34:38.760680 26537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f1dbdf3e534160b07f9f4d5422d153" candidate_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e6e93f89c754a66a9a375ed9d062694"
I20260925 02:34:38.760746 26537 raft_consensus.cc:3037] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:38.760746 26669 raft_consensus.cc:3037] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:34:38.761659 26537 raft_consensus.cc:2454] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5a6c1c0ccf49f3a7a59126c1745985 in term 1.
I20260925 02:34:38.761659 26669 raft_consensus.cc:2454] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5a6c1c0ccf49f3a7a59126c1745985 in term 1.
I20260925 02:34:38.761879 26332 leader_election.cc:302] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5a6c1c0ccf49f3a7a59126c1745985, 8e6e93f89c754a66a9a375ed9d062694; no voters:
I20260925 02:34:38.761999 26451 raft_consensus.cc:2781] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:34:38.762166 26451 raft_consensus.cc:686] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5a6c1c0ccf49f3a7a59126c1745985, State: Running, Role: LEADER
I20260925 02:34:38.762270 26451 consensus_queue.cc:237] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:38.763082 26257 catalog_manager.cc:5495] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a5a6c1c0ccf49f3a7a59126c1745985 (127.10.13.1). New cstate: current_term: 1 leader_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } health_report { overall_health: HEALTHY } } }
I20260925 02:34:38.778581 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.776179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.10.13.1:37881)
0925 02:34:38.776262 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 02:34:38.776264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:34:38.776272 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:34:38.776310 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:34:38.776311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:34:38.776330 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:34:38.776392 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:34:38.776554 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.777179 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.777883 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:34:38.778097 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:34:38.778161 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:34:38.778295 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:34:38.778399 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:34:38.778457 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:34:38.778494 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 02:34:38.778514 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:34:38.780289 26669 raft_consensus.cc:1261] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 1 FOLLOWER]: Refusing update from remote peer 8a5a6c1c0ccf49f3a7a59126c1745985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:34:38.780289 26537 raft_consensus.cc:1261] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 FOLLOWER]: Refusing update from remote peer 8a5a6c1c0ccf49f3a7a59126c1745985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:34:38.780730 26732 consensus_queue.cc:1035] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:34:38.780903 26451 consensus_queue.cc:1035] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:34:38.783457 26735 mvcc.cc:205] Tried to move back new op lower bound from 7333083868281073664 to 7333083868210827264. Current Snapshot: MvccSnapshot[applied={T|T < 7333083868281073664}]
I20260925 02:34:38.784011 26737 mvcc.cc:205] Tried to move back new op lower bound from 7333083868281073664 to 7333083868210827264. Current Snapshot: MvccSnapshot[applied={T|T < 7333083868281073664}]
I20260925 02:34:38.783792 26736 mvcc.cc:205] Tried to move back new op lower bound from 7333083868281073664 to 7333083868210827264. Current Snapshot: MvccSnapshot[applied={T|T < 7333083868281073664}]
I20260925 02:34:38.786057 26257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:34:38.786161 26257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:34:38.786993 26257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Sending DeleteTablet for 3 replicas of tablet e8f1dbdf3e534160b07f9f4d5422d153
I20260925 02:34:38.787389 26517 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f1dbdf3e534160b07f9f4d5422d153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:38 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:45244
I20260925 02:34:38.787396 26649 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f1dbdf3e534160b07f9f4d5422d153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:38 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:35290
I20260925 02:34:38.787627 26744 tablet_replica.cc:330] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: stopping tablet replica
I20260925 02:34:38.787735 26366 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f1dbdf3e534160b07f9f4d5422d153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:34:38 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:56476
I20260925 02:34:38.787781 26745 tablet_replica.cc:330] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: stopping tablet replica
I20260925 02:34:38.787817 26454 tablet_replica.cc:330] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: stopping tablet replica
I20260925 02:34:38.787822 26744 raft_consensus.cc:2229] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:34:38.787859 26454 raft_consensus.cc:2229] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:34:38.787878 26745 raft_consensus.cc:2229] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:34:38.787936 26454 raft_consensus.cc:2258] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:34:38.787946 26744 raft_consensus.cc:2258] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:34:38.787977 26745 raft_consensus.cc:2258] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:34:38.788319 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 26295
I20260925 02:34:38.788430 26745 ts_tablet_manager.cc:1905] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:34:38.788436 26744 ts_tablet_manager.cc:1905] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:34:38.790515 26745 ts_tablet_manager.cc:1918] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:34:38.790540 26744 ts_tablet_manager.cc:1918] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:34:38.790609 26744 log.cc:1196] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/e8f1dbdf3e534160b07f9f4d5422d153
I20260925 02:34:38.790666 26745 log.cc:1196] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/e8f1dbdf3e534160b07f9f4d5422d153
I20260925 02:34:38.791016 26745 ts_tablet_manager.cc:1939] T e8f1dbdf3e534160b07f9f4d5422d153 P 16ca5b0694ac4269ab124c10564c88e6: Deleting consensus metadata
I20260925 02:34:38.791042 26744 ts_tablet_manager.cc:1939] T e8f1dbdf3e534160b07f9f4d5422d153 P 8e6e93f89c754a66a9a375ed9d062694: Deleting consensus metadata
I20260925 02:34:38.791473 26242 catalog_manager.cc:4841] TS 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469): tablet e8f1dbdf3e534160b07f9f4d5422d153 (table test-table [id=4031565db0964396a9aa47b7b4519c6b]) successfully deleted
I20260925 02:34:38.791594 26242 catalog_manager.cc:4841] TS 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377): tablet e8f1dbdf3e534160b07f9f4d5422d153 (table test-table [id=4031565db0964396a9aa47b7b4519c6b]) successfully deleted
W20260925 02:34:38.794911 26244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:37881 (error 108)
I20260925 02:34:38.795172 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 26450
I20260925 02:34:38.795415 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:38.795283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37881 (local address 127.0.0.1:56508)
0925 02:34:38.795331 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:38.795362 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 02:34:38.795539 26244 catalog_manager.cc:4584] TS 8a5a6c1c0ccf49f3a7a59126c1745985 (127.10.13.1:37881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8f1dbdf3e534160b07f9f4d5422d153: Network error: Client connection negotiation failed: client connection to 127.10.13.1:37881: connect: Connection refused (error 111)
I20260925 02:34:38.800608 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 26586
I20260925 02:34:38.805321 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 26224
I20260925 02:34:58.811215 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39519
--webserver_interface=127.10.13.62
--webserver_port=44385
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:39519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:58.896505 26747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:58.896719 26747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:58.896742 26747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:58.897980 26747 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:34:58.898025 26747 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:34:58.898038 26747 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:34:58.898051 26747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:34:58.898101 26747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:58.898128 26747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:58.898154 26747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:34:58.898178 26747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:34:58.899718 26747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:39519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:39519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:58.900002 26747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:58.900241 26747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:58.905555 26755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:58.905701 26747 server_base.cc:1034] running on GCE node
W20260925 02:34:58.905567 26752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:58.905608 26753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:58.906141 26747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:58.906517 26747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:58.907689 26747 hybrid_clock.cc:648] HybridClock initialized: now 1790303698907668 us; error 37 us; skew 500 ppm
Sep 25 02:34:58 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303698, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:58.918450 26747 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:34:58.919003 26747 webserver.cc:460] Webserver started at http://127.10.13.62:44385/ using document root <none> and password file <none>
I20260925 02:34:58.919266 26747 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:58.919322 26747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:58.920781 26747 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:34:58.921541 26762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:58.921710 26747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:34:58.921821 26747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "e0ed57e3d44c453a9778129e3005c039"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "60f04478e590c5ec6a9de6066a740c1e"
server_key_iv: "93bbb3e765c3b6e35e7a206a8dd41431"
server_key_version: "encryptionkey@0"
I20260925 02:34:58.922190 26747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:58.964393 26747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:58.967206 26747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:58.967393 26747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:58.971783 26747 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:39519
I20260925 02:34:58.971848 26814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:39519 every 8 connection(s)
I20260925 02:34:58.972247 26747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:34:58.973883 26815 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:34:58.974493 26815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Bootstrap starting.
I20260925 02:34:58.977485 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 26747
I20260925 02:34:58.977818 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:37881
--local_ip_for_outbound_sockets=127.10.13.1
--tserver_master_addrs=127.10.13.62:39519
--webserver_port=33281
--webserver_interface=127.10.13.1
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:34:58.983597 26815 log.cc:824] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:58.986132 26815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:34:58.986526 26815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Bootstrap complete.
I20260925 02:34:58.989030 26815 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:58.989264 26815 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ed57e3d44c453a9778129e3005c039, State: Initialized, Role: FOLLOWER
I20260925 02:34:58.989434 26815 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:58.989494 26815 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:34:58.989567 26815 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:34:58.989619 26815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:34:58.990651 26815 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:58.990768 26815 leader_election.cc:302] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ed57e3d44c453a9778129e3005c039; no voters:
I20260925 02:34:58.990988 26815 leader_election.cc:288] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:34:58.991026 26819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:34:58.991156 26819 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [term 3 LEADER]: Becoming Leader. State: Replica: e0ed57e3d44c453a9778129e3005c039, State: Running, Role: LEADER
I20260925 02:34:58.991286 26815 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:34:58.991325 26819 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } }
I20260925 02:34:58.991631 26819 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ed57e3d44c453a9778129e3005c039. Latest consensus state: current_term: 3 leader_uuid: "e0ed57e3d44c453a9778129e3005c039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } } }
I20260925 02:34:58.991703 26819 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: This master's current role is: LEADER
I20260925 02:34:58.991653 26820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e0ed57e3d44c453a9778129e3005c039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ed57e3d44c453a9778129e3005c039" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 39519 } } }
I20260925 02:34:58.991894 26820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039 [sys.catalog]: This master's current role is: LEADER
I20260925 02:34:58.992125 26824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:34:58.993141 26824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:34:58.993331 26824 catalog_manager.cc:1259] Loaded cluster ID: 0dd16d79f7f346669130e09ad68b9354
I20260925 02:34:58.993551 26824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:34:58.994395 26824 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:34:58.994585 26824 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Loaded TSK: 0
I20260925 02:34:58.994645 26824 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0ed57e3d44c453a9778129e3005c039: Loaded TSK: 1
F20260925 02:34:58.994866 26824 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1a271589ad google::LogMessage::Fail() at ??:0
@ 0x7f1a2715b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1a271584de google::LogMessage::Flush() at ??:0
@ 0x7f1a27159a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1a298c5104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1a298c2d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1a2f0b21cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1a2f0b248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1a2f0b2594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1a2f078c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1a2f082ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1a290260d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1a2901b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1a29e7f6db start_thread at ??:0
@ 0x7f1a25c5661f clone at ??:0
W20260925 02:34:59.100514 26817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:59.100821 26817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:59.100858 26817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:59.102586 26817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:59.102653 26817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:59.102695 26817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:34:59.104779 26817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:37881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=33281
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:59.105060 26817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:59.105352 26817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:59.110704 26856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:59.110718 26859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:59.110895 26817 server_base.cc:1034] running on GCE node
W20260925 02:34:59.111006 26857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:59.111277 26817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:59.111538 26817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:59.112715 26817 hybrid_clock.cc:648] HybridClock initialized: now 1790303699112696 us; error 33 us; skew 500 ppm
Sep 25 02:34:59 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303699, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:59.115378 26817 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:34:59.115820 26817 webserver.cc:460] Webserver started at http://127.10.13.1:33281/ using document root <none> and password file <none>
I20260925 02:34:59.116070 26817 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:59.116137 26817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:59.117795 26817 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:34:59.118572 26866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:59.118820 26817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:34:59.118944 26817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "8a5a6c1c0ccf49f3a7a59126c1745985"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "ff66d5bca9bbc0f9ff32fc038141b884"
server_key_iv: "bd44e2ea208d052c757f2c38797caedc"
server_key_version: "encryptionkey@0"
I20260925 02:34:59.119341 26817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:59.143442 26817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:59.146099 26817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:59.146286 26817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:59.146564 26817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:59.147074 26873 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:34:59.149189 26817 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:34:59.149286 26817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:34:59.149335 26817 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:34:59.150056 26817 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:34:59.150111 26817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:34:59.150187 26873 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Bootstrap starting.
I20260925 02:34:59.157510 26817 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:37881
I20260925 02:34:59.157557 26980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:37881 every 8 connection(s)
I20260925 02:34:59.157963 26817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:34:59.162086 26873 log.cc:824] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Log is configured to *not* fsync() on all Append() calls
I20260925 02:34:59.165126 26873 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:34:59.165457 26873 tablet_bootstrap.cc:492] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Bootstrap complete.
I20260925 02:34:59.165704 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 26817
I20260925 02:34:59.166049 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:38377
--local_ip_for_outbound_sockets=127.10.13.2
--tserver_master_addrs=127.10.13.62:39519
--webserver_port=37561
--webserver_interface=127.10.13.2
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:34:59.166769 26873 ts_tablet_manager.cc:1397] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260925 02:34:59.168321 26873 raft_consensus.cc:348] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:59.168684 26873 raft_consensus.cc:729] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5a6c1c0ccf49f3a7a59126c1745985, State: Initialized, Role: FOLLOWER
I20260925 02:34:59.168896 26873 consensus_queue.cc:260] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:34:59.169219 26873 ts_tablet_manager.cc:1428] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260925 02:34:59.220963 26983 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:34:59.158863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:34645)
0925 02:34:59.159096 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.159139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:34:59.159667 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:34:59.220612 (+ 60945us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
W20260925 02:34:59.221206 26981 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:39519 (0 consecutive failures): Network error: Failed to ping master at 127.10.13.62:39519: Client connection negotiation failed: client connection to 127.10.13.62:39519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:34:59.221622 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:59.221497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:43601)
0925 02:34:59.221549 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.221593 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:34:59.221972 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:59.221866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:52005)
0925 02:34:59.221924 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.221948 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:34:59.222090 26981 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:34:59.256150 26985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:59.256405 26985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:59.256451 26985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:59.257711 26985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:59.257779 26985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:59.257825 26985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:34:59.259330 26985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:38377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=37561
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:59.259608 26985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:59.259876 26985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:34:59.264837 26994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:59.264860 26991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:59.264946 26992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:59.265130 26985 server_base.cc:1034] running on GCE node
I20260925 02:34:59.265344 26985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:59.265602 26985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:59.266769 26985 hybrid_clock.cc:648] HybridClock initialized: now 1790303699266762 us; error 36 us; skew 500 ppm
Sep 25 02:34:59 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303699, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:59.269214 26985 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:34:59.269641 26985 webserver.cc:460] Webserver started at http://127.10.13.2:37561/ using document root <none> and password file <none>
I20260925 02:34:59.269891 26985 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:59.269959 26985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:59.271286 26985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:59.271986 27001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:59.272176 26985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:59.272279 26985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "8e6e93f89c754a66a9a375ed9d062694"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "3ae6389f1628fe929157dd5b77481ee9"
server_key_iv: "101727cea129e3e2c49b3d345e1f1cd4"
server_key_version: "encryptionkey@0"
I20260925 02:34:59.272622 26985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:59.283207 26985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:59.285434 26985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:59.285596 26985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:59.285830 26985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:59.286146 26985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:59.286197 26985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:59.286239 26985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:59.286278 26985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:59.292173 26985 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:38377
I20260925 02:34:59.292249 27114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:38377 every 8 connection(s)
I20260925 02:34:59.292623 26985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:34:59.293792 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:59.293400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:52369)
0925 02:34:59.293559 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.293635 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
W20260925 02:34:59.294219 27115 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:39519 (0 consecutive failures): Network error: Failed to ping master at 127.10.13.62:39519: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
I20260925 02:34:59.294492 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:59.294404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:52391)
0925 02:34:59.294446 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.294473 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 02:34:59.294811 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:59.294686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:48669)
0925 02:34:59.294749 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.294784 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:34:59.294978 27115 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:34:59.302075 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 26985
I20260925 02:34:59.302395 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:38469
--local_ip_for_outbound_sockets=127.10.13.3
--tserver_master_addrs=127.10.13.62:39519
--webserver_port=37393
--webserver_interface=127.10.13.3
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:34:59.388386 27118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:34:59.388607 27118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:34:59.388634 27118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:34:59.389863 27118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:34:59.389905 27118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:34:59.389931 27118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:34:59.391424 27118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:38469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=37393
--tserver_master_addrs=127.10.13.62:39519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:34:59.391701 27118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:34:59.391947 27118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:34:59.397631 27118 server_base.cc:1034] running on GCE node
W20260925 02:34:59.397614 27126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:59.397614 27123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:34:59.397593 27124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:34:59.398023 27118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:34:59.398290 27118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:34:59.399449 27118 hybrid_clock.cc:648] HybridClock initialized: now 1790303699399424 us; error 35 us; skew 500 ppm
Sep 25 02:34:59 dist-test-slave-q8mg krb5kdc[25676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303699, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:34:59.402217 27118 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:34:59.402714 27118 webserver.cc:460] Webserver started at http://127.10.13.3:37393/ using document root <none> and password file <none>
I20260925 02:34:59.402964 27118 fs_manager.cc:359] Metadata directory not provided
I20260925 02:34:59.403012 27118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:34:59.404421 27118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:59.405150 27133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:59.405318 27118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:34:59.405426 27118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "16ca5b0694ac4269ab124c10564c88e6"
format_stamp: "Formatted at 2026-09-25 02:34:17 on dist-test-slave-q8mg"
server_key: "a6a9b2fecd8f2a61918d74d23012226d"
server_key_iv: "39e04a083137b0ced24bce53bbacbfd9"
server_key_version: "encryptionkey@0"
I20260925 02:34:59.405827 27118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:34:59.420409 27118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:34:59.422562 27118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:34:59.422722 27118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:34:59.422943 27118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:34:59.423267 27118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:34:59.423301 27118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:59.423324 27118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:34:59.423368 27118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:34:59.429332 27118 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:38469
I20260925 02:34:59.429390 27246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:38469 every 8 connection(s)
I20260925 02:34:59.429731 27118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:34:59.430846 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:59.430458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:58065)
0925 02:34:59.430628 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.430699 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
W20260925 02:34:59.431270 27247 heartbeater.cc:644] Failed to heartbeat to 127.10.13.62:39519 (0 consecutive failures): Network error: Failed to ping master at 127.10.13.62:39519: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
I20260925 02:34:59.431531 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:59.431446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:33495)
0925 02:34:59.431487 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.431514 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 02:34:59.431802 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:34:59.431733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:51313)
0925 02:34:59.431769 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 02:34:59.431790 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 02:34:59.431918 27247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:34:59.437947 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 27118
I20260925 02:35:00.222796 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:00.222531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:35741)
0925 02:35:00.222672 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:00.222715 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260925 02:35:00.295588 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:00.295353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:40525)
0925 02:35:00.295470 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:00.295503 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260925 02:35:00.432572 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:00.432318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:36521)
0925 02:35:00.432450 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:00.432496 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260925 02:35:00.933737 27254 raft_consensus.cc:482] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:00.933871 27254 raft_consensus.cc:504] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:35:00.934255 27254 leader_election.cc:288] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469), 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377)
Sep 25 02:35:00 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303699, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.2@KRBTEST.COM
Sep 25 02:35:00 dist-test-slave-q8mg krb5kdc[25676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303699, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.3@KRBTEST.COM
I20260925 02:35:00.945657 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:00.934767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:38329 (local address 127.10.13.2:38377)
0925 02:35:00.934909 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:00.934920 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:00.934947 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:35:00.935424 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:00.935428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:00.935483 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:00.935638 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:00.936267 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:00.937300 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:00.938296 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:00.938740 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:00.938891 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:00.941384 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:00.941388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:00.941388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:00.941398 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:00.943431 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:00.944105 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:00.944109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:00.944109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:00.944227 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:00.944536 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:00.944537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:00.944537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:00.944901 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:00.944991 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:00.945399 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 02:35:00.945536 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260925 02:35:00.945657 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:00.934552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:59757 (local address 127.10.13.3:38469)
0925 02:35:00.934735 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:00.934749 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:00.934777 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:35:00.935229 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:00.935231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:00.935284 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:00.935395 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:00.936296 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:00.937191 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:00.938344 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:00.938730 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:00.938841 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:00.941641 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:00.941647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:00.941647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:00.941659 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:00.943455 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:00.944105 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:00.944109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:00.944109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:00.944227 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:00.944536 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:00.944537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:00.944537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:00.944929 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:00.945003 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:00.945398 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 02:35:00.945536 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":53,"threads_started":1}
I20260925 02:35:00.945696 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:00.934621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:38377 (local address 127.10.13.1:38329)
0925 02:35:00.934710 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:00.934720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:00.934785 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:00.935622 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:00.935623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:00.935937 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:00.936142 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:00.936148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:00.937392 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:00.937393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:00.938207 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:00.938209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:00.938853 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:00.938854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:00.938944 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:00.939591 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:00.939595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:00.941270 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:00.943537 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:00.943547 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:00.943549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:00.944010 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:00.944311 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:00.944312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:00.944312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:00.944468 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:00.945004 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:00.945014 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:00.945181 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 02:35:00.945322 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 02:35:00.945548 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":62,"thread_start_us":33,"threads_started":1}
I20260925 02:35:00.945704 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:00.934419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:38469 (local address 127.10.13.1:59757)
0925 02:35:00.934549 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:00.934561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:00.934649 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:00.935487 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:00.935491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:00.935939 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:00.936143 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:00.936149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:00.937294 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:00.937296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:00.938209 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:00.938212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:00.938820 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:00.938821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:00.938946 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:00.939533 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:00.939542 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:00.941531 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:00.943537 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:00.943547 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:00.943549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:00.944010 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:00.944304 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:00.944305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:00.944305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:00.944468 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:00.945003 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:00.945014 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:00.945181 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 02:35:00.945322 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 02:35:00.945604 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":202,"thread_start_us":90,"threads_started":1}
W20260925 02:35:00.946959 26868 leader_election.cc:341] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469): Not found: Tablet not found: e8f1dbdf3e534160b07f9f4d5422d153
W20260925 02:35:00.947042 26868 leader_election.cc:341] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377): Not found: Tablet not found: e8f1dbdf3e534160b07f9f4d5422d153
I20260925 02:35:00.947088 26868 leader_election.cc:302] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a5a6c1c0ccf49f3a7a59126c1745985; no voters: 16ca5b0694ac4269ab124c10564c88e6, 8e6e93f89c754a66a9a375ed9d062694
I20260925 02:35:00.947221 27254 raft_consensus.cc:2726] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:35:01.223476 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:01.223327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:60391)
0925 02:35:01.223383 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:01.223432 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:35:01.296384 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:01.296106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:54035)
0925 02:35:01.296268 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:01.296333 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260925 02:35:01.433362 27260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:01.433108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:50279)
0925 02:35:01.433251 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:01.433309 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260925 02:35:02.224139 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:02.223909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:51011)
0925 02:35:02.224048 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:02.224090 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260925 02:35:02.297216 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:02.297014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:36439)
0925 02:35:02.297133 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:02.297172 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260925 02:35:02.434154 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:02.433931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:55353)
0925 02:35:02.434062 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:02.434103 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260925 02:35:02.453122 27264 raft_consensus.cc:482] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:02.453218 27264 raft_consensus.cc:504] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:35:02.453418 27264 leader_election.cc:288] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469), 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377)
W20260925 02:35:02.453961 26868 leader_election.cc:341] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377): Not found: Tablet not found: e8f1dbdf3e534160b07f9f4d5422d153
W20260925 02:35:02.454049 26868 leader_election.cc:341] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469): Not found: Tablet not found: e8f1dbdf3e534160b07f9f4d5422d153
I20260925 02:35:02.454085 26868 leader_election.cc:302] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a5a6c1c0ccf49f3a7a59126c1745985; no voters: 16ca5b0694ac4269ab124c10564c88e6, 8e6e93f89c754a66a9a375ed9d062694
I20260925 02:35:02.454182 27264 raft_consensus.cc:2726] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:35:03.224977 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:03.224656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:45793)
0925 02:35:03.224809 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:03.224855 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260925 02:35:03.297902 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:03.297678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:45757)
0925 02:35:03.297815 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:03.297858 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260925 02:35:03.435005 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:03.434718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:53399)
0925 02:35:03.434899 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:03.434958 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260925 02:35:04.225745 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:04.225517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:33905)
0925 02:35:04.225657 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:04.225695 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260925 02:35:04.298583 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:04.298365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:55077)
0925 02:35:04.298498 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:04.298539 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260925 02:35:04.435875 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:04.435585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:60813)
0925 02:35:04.435775 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:04.435823 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260925 02:35:04.528436 27271 raft_consensus.cc:482] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:04.528538 27271 raft_consensus.cc:504] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:35:04.528677 27271 leader_election.cc:288] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469), 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377)
W20260925 02:35:04.529211 26868 leader_election.cc:341] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469): Not found: Tablet not found: e8f1dbdf3e534160b07f9f4d5422d153
W20260925 02:35:04.529313 26868 leader_election.cc:341] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377): Not found: Tablet not found: e8f1dbdf3e534160b07f9f4d5422d153
I20260925 02:35:04.529350 26868 leader_election.cc:302] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a5a6c1c0ccf49f3a7a59126c1745985; no voters: 16ca5b0694ac4269ab124c10564c88e6, 8e6e93f89c754a66a9a375ed9d062694
I20260925 02:35:04.529462 27271 raft_consensus.cc:2726] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:35:05.226478 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:05.226236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:58115)
0925 02:35:05.226378 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:05.226426 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260925 02:35:05.299425 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:05.299136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:58061)
0925 02:35:05.299288 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:05.299357 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260925 02:35:05.436688 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:05.436452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:55339)
0925 02:35:05.436585 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:05.436632 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260925 02:35:06.227167 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:06.226962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:60465)
0925 02:35:06.227085 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:06.227119 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260925 02:35:06.300191 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:06.299993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:39103)
0925 02:35:06.300110 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:06.300146 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260925 02:35:06.437407 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:06.437194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:35973)
0925 02:35:06.437316 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:06.437350 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260925 02:35:07.228020 27278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:07.227754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:37589)
0925 02:35:07.227924 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:07.227970 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260925 02:35:07.301048 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:07.300753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:41095)
0925 02:35:07.300955 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:07.301000 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260925 02:35:07.438131 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:07.437906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:44619)
0925 02:35:07.438039 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:07.438083 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260925 02:35:07.510702 27281 raft_consensus.cc:482] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:07.510803 27281 raft_consensus.cc:504] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ca5b0694ac4269ab124c10564c88e6" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38469 } } peers { permanent_uuid: "8e6e93f89c754a66a9a375ed9d062694" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 38377 } } peers { permanent_uuid: "8a5a6c1c0ccf49f3a7a59126c1745985" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37881 } }
I20260925 02:35:07.510986 27281 leader_election.cc:288] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469), 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377)
W20260925 02:35:07.511382 26868 leader_election.cc:341] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e6e93f89c754a66a9a375ed9d062694 (127.10.13.2:38377): Not found: Tablet not found: e8f1dbdf3e534160b07f9f4d5422d153
W20260925 02:35:07.511467 26868 leader_election.cc:341] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16ca5b0694ac4269ab124c10564c88e6 (127.10.13.3:38469): Not found: Tablet not found: e8f1dbdf3e534160b07f9f4d5422d153
I20260925 02:35:07.511482 26868 leader_election.cc:302] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a5a6c1c0ccf49f3a7a59126c1745985; no voters: 16ca5b0694ac4269ab124c10564c88e6, 8e6e93f89c754a66a9a375ed9d062694
I20260925 02:35:07.511552 27281 raft_consensus.cc:2726] T e8f1dbdf3e534160b07f9f4d5422d153 P 8a5a6c1c0ccf49f3a7a59126c1745985 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:35:08.228763 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:08.228526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:41019)
0925 02:35:08.228663 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:08.228713 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260925 02:35:08.301783 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:08.301569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:54097)
0925 02:35:08.301696 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:08.301735 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260925 02:35:08.439000 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:08.438683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:53073)
0925 02:35:08.438904 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:08.438942 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":155,"threads_started":1}
I20260925 02:35:09.229625 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:09.229338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:33683)
0925 02:35:09.229509 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:09.229549 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260925 02:35:09.302560 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:09.302292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:37497)
0925 02:35:09.302466 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:09.302510 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
I20260925 02:35:09.439795 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:09.439542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:57343)
0925 02:35:09.439700 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:09.439745 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260925 02:35:10.230463 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:10.230198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:45803)
0925 02:35:10.230367 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:10.230412 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260925 02:35:10.303401 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:10.303185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:52119)
0925 02:35:10.303309 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:10.303349 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260925 02:35:10.440654 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:10.440372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:59311)
0925 02:35:10.440559 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:10.440604 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20260925 02:35:11.231205 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:11.230983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:49141)
0925 02:35:11.231116 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:11.231157 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260925 02:35:11.304131 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:11.303911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:37359)
0925 02:35:11.304041 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:11.304082 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260925 02:35:11.441407 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:11.441181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:33783)
0925 02:35:11.441318 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:11.441359 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260925 02:35:12.231904 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:12.231691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:34177)
0925 02:35:12.231816 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:12.231849 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260925 02:35:12.304983 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:12.304697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:39635)
0925 02:35:12.304842 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:12.304925 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260925 02:35:12.442332 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:12.442067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:39281)
0925 02:35:12.442246 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:12.442279 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":115,"threads_started":1}
I20260925 02:35:13.232698 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:13.232447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:41819)
0925 02:35:13.232603 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:13.232646 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260925 02:35:13.305720 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:13.305504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:59917)
0925 02:35:13.305628 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:13.305670 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260925 02:35:13.443172 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:13.442901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:47845)
0925 02:35:13.443071 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:13.443119 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260925 02:35:14.233593 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:14.233366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.1:55167)
0925 02:35:14.233503 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:14.233543 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260925 02:35:14.306517 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:14.306288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.2:49327)
0925 02:35:14.306431 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:14.306469 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260925 02:35:14.441574 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 26817
I20260925 02:35:14.444011 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:14.443782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:39519 (local address 127.10.13.3:57291)
0925 02:35:14.443907 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:14.443967 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.62:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":86,"threads_started":1}
I20260925 02:35:14.447176 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 26985
I20260925 02:35:14.451867 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 27118
2026-09-25T02:35:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58532 ms)
[----------] 30 tests from SecurityITest (235865 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:14 dist-test-slave-q8mg krb5kdc[27305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:14 dist-test-slave-q8mg krb5kdc[27305](info): set up 2 sockets
Sep 25 02:35:14 dist-test-slave-q8mg krb5kdc[27305](info): commencing operation
krb5kdc: starting...
W20260925 02:35:15.495184 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:15 dist-test-slave-q8mg krb5kdc[27305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:15.621222 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:43605
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:43605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:15.704041 27318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:15.704252 27318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:15.704276 27318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:15.705449 27318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:15.705490 27318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:15.705504 27318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:15.705518 27318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:15.705562 27318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:15.707050 27318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:43605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:43605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:15.707309 27318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:15.707533 27318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:15.712800 27323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:15.712827 27324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:15.712831 27326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:15.712906 27318 server_base.cc:1034] running on GCE node
I20260925 02:35:15.713364 27318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:15.713671 27318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:15.714841 27318 hybrid_clock.cc:648] HybridClock initialized: now 1790303715714839 us; error 40 us; skew 500 ppm
Sep 25 02:35:15 dist-test-slave-q8mg krb5kdc[27305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303715, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:15.722570 27318 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:15.723057 27318 webserver.cc:460] Webserver started at http://127.10.13.62:35821/ using document root <none> and password file <none>
I20260925 02:35:15.723322 27318 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:15.723398 27318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:15.723567 27318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:15.724686 27318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "25cf8a789db047d7aa7c748b36f5a265"
format_stamp: "Formatted at 2026-09-25 02:35:15 on dist-test-slave-q8mg"
server_key: "922afe6efc42938adfc42698205a39b7"
server_key_iv: "20159b14fc86e75dd3eb90bdfa618709"
server_key_version: "encryptionkey@0"
I20260925 02:35:15.725098 27318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "25cf8a789db047d7aa7c748b36f5a265"
format_stamp: "Formatted at 2026-09-25 02:35:15 on dist-test-slave-q8mg"
server_key: "922afe6efc42938adfc42698205a39b7"
server_key_iv: "20159b14fc86e75dd3eb90bdfa618709"
server_key_version: "encryptionkey@0"
I20260925 02:35:15.726506 27318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:35:15.727267 27333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:15.727454 27318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:35:15.727568 27318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "25cf8a789db047d7aa7c748b36f5a265"
format_stamp: "Formatted at 2026-09-25 02:35:15 on dist-test-slave-q8mg"
server_key: "922afe6efc42938adfc42698205a39b7"
server_key_iv: "20159b14fc86e75dd3eb90bdfa618709"
server_key_version: "encryptionkey@0"
I20260925 02:35:15.727645 27318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:15.744259 27318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:15.746363 27318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:15.746547 27318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:15.750779 27318 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:43605
I20260925 02:35:15.750847 27385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:43605 every 8 connection(s)
I20260925 02:35:15.751214 27318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:15.752138 27386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:15.754500 27386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265: Bootstrap starting.
I20260925 02:35:15.755043 27386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:15.755280 27386 log.cc:824] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:15.755956 27386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265: No bootstrap required, opened a new log
I20260925 02:35:15.756300 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 27318
I20260925 02:35:15.756428 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:15.756624 10292 external_mini_cluster.cc:1428] Setting key b800d444d668b9a0f5ee0cb20a70139d
I20260925 02:35:15.758375 27386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf8a789db047d7aa7c748b36f5a265" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43605 } }
I20260925 02:35:15.758507 27386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:15.758529 27386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cf8a789db047d7aa7c748b36f5a265, State: Initialized, Role: FOLLOWER
I20260925 02:35:15.758657 27386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf8a789db047d7aa7c748b36f5a265" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43605 } }
I20260925 02:35:15.758733 27386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:15.758755 27386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:15.758822 27386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:15.759392 27386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf8a789db047d7aa7c748b36f5a265" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43605 } }
I20260925 02:35:15.759512 27386 leader_election.cc:302] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25cf8a789db047d7aa7c748b36f5a265; no voters:
I20260925 02:35:15.759713 27386 leader_election.cc:288] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:15.759737 27391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:15.759858 27391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [term 1 LEADER]: Becoming Leader. State: Replica: 25cf8a789db047d7aa7c748b36f5a265, State: Running, Role: LEADER
I20260925 02:35:15.759996 27386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:15.760006 27391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf8a789db047d7aa7c748b36f5a265" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43605 } }
I20260925 02:35:15.760406 27392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25cf8a789db047d7aa7c748b36f5a265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf8a789db047d7aa7c748b36f5a265" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43605 } } }
I20260925 02:35:15.760427 27393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25cf8a789db047d7aa7c748b36f5a265. Latest consensus state: current_term: 1 leader_uuid: "25cf8a789db047d7aa7c748b36f5a265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf8a789db047d7aa7c748b36f5a265" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43605 } } }
I20260925 02:35:15.760659 27392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:15.760675 27393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:15.761485 27400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:15.762238 27400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:35:15 dist-test-slave-q8mg krb5kdc[27305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:15.765359 27400 catalog_manager.cc:1347] Generated new cluster ID: bc4e22593ff64bb5ae6ce61fb4895035
I20260925 02:35:15.765412 27400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:15.768399 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:15.757216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.10.13.62:43605)
0925 02:35:15.757425 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:15.757435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:15.757470 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:35:15.757950 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:15.757959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:15.758025 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:15.758184 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:15.758878 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:15.760361 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:15.761507 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:15.762087 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:15.762206 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:15.765019 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:15.765023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:15.765023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:15.765036 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:15.766830 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:15.767222 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:15.767225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:15.767225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:15.767336 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:15.767552 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:15.767553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:15.767554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:15.767808 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:15.767982 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:15.768085 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 02:35:15.768231 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260925 02:35:15.789182 27400 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:15.789770 27400 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:15.792099 27400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265: Generated new TSK 0
I20260925 02:35:15.792289 27400 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:35:15.826167 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43605
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:15.910219 27412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:15.910419 27412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:15.910445 27412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:15.911681 27412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:15.911725 27412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:15.911751 27412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:35:15.913290 27412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:15.913547 27412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:15.913789 27412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:15.919065 27417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:15.919059 27418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:15.919116 27412 server_base.cc:1034] running on GCE node
W20260925 02:35:15.919036 27420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:15.919461 27412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:15.919670 27412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:15.920822 27412 hybrid_clock.cc:648] HybridClock initialized: now 1790303715920799 us; error 38 us; skew 500 ppm
Sep 25 02:35:15 dist-test-slave-q8mg krb5kdc[27305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303715, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:15.923456 27412 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:35:15.923969 27412 webserver.cc:460] Webserver started at http://127.10.13.1:39147/ using document root <none> and password file <none>
I20260925 02:35:15.924218 27412 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:15.924295 27412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:15.924454 27412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:15.925593 27412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "74676206e6454f6e9fc43a082b207ee7"
format_stamp: "Formatted at 2026-09-25 02:35:15 on dist-test-slave-q8mg"
server_key: "861d262691769fb9e1f6ea35541e2bc1"
server_key_iv: "96359906e21a25d978639923d95db35c"
server_key_version: "encryptionkey@0"
I20260925 02:35:15.925973 27412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "74676206e6454f6e9fc43a082b207ee7"
format_stamp: "Formatted at 2026-09-25 02:35:15 on dist-test-slave-q8mg"
server_key: "861d262691769fb9e1f6ea35541e2bc1"
server_key_iv: "96359906e21a25d978639923d95db35c"
server_key_version: "encryptionkey@0"
I20260925 02:35:15.927366 27412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:35:15.928188 27427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:15.928371 27412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:15.928485 27412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "74676206e6454f6e9fc43a082b207ee7"
format_stamp: "Formatted at 2026-09-25 02:35:15 on dist-test-slave-q8mg"
server_key: "861d262691769fb9e1f6ea35541e2bc1"
server_key_iv: "96359906e21a25d978639923d95db35c"
server_key_version: "encryptionkey@0"
I20260925 02:35:15.928565 27412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:15.939970 27412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:15.942147 27412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:15.942313 27412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:15.942553 27412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:15.942889 27412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:15.942945 27412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:15.942986 27412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:15.943012 27412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:15.948963 27412 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:33763
I20260925 02:35:15.949028 27540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:33763 every 8 connection(s)
I20260925 02:35:15.949384 27412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:35:15.951359 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 27412
I20260925 02:35:15.951448 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:35:15.951686 10292 external_mini_cluster.cc:1428] Setting key ac370c0cbb5cb593cbdcc01f7e3401eb
Sep 25 02:35:15 dist-test-slave-q8mg krb5kdc[27305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303715, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:15.960438 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:15.950172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:48731 (local address 127.10.13.62:43605)
0925 02:35:15.950237 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:15.950239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:15.950466 (+ 227us) server_negotiation.cc:407] Connection header received
0925 02:35:15.951000 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:15.951002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:15.951015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:15.951073 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:15.952252 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:15.952839 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:15.954355 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:15.954589 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:15.954680 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:15.957251 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:15.957252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:15.957253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:15.957262 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:15.958945 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:15.959355 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:15.959356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:15.959357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:15.959424 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:15.959669 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:15.959671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:15.959671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:15.959817 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:15.959893 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:15.960214 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:35:15.960315 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:35:15.960513 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:15.950138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43605 (local address 127.10.13.1:48731)
0925 02:35:15.950335 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:15.950376 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:15.950890 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:15.951153 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:15.951156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:15.951898 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:15.952159 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:15.952165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:15.952989 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:15.952990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:15.954259 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:15.954261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:15.954695 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:15.954697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:15.954851 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:15.955387 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:15.955399 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:15.957147 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:15.959031 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:15.959038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:15.959040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:15.959274 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:15.959498 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:15.959499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:15.959499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:15.959600 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:15.959889 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:15.959895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:15.960054 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 02:35:15.960168 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:35:15.960302 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260925 02:35:15.961149 27541 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43605
I20260925 02:35:15.961254 27541 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
I20260925 02:35:15.961467 27541 heartbeater.cc:511] Master 127.10.13.62:43605 requested a full tablet report, sending...
I20260925 02:35:15.962069 27350 ts_manager.cc:194] Registered new tserver with Master: 74676206e6454f6e9fc43a082b207ee7 (127.10.13.1:33763)
I20260925 02:35:15.962800 27350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:48731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:35:15.977944 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43605
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:16.060194 27546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:16.060469 27546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:16.060513 27546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:16.061851 27546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:16.061919 27546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:16.061966 27546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:35:16.063495 27546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:16.063777 27546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:16.064028 27546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:16.069202 27554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.069202 27551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.069212 27552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.069479 27546 server_base.cc:1034] running on GCE node
I20260925 02:35:16.069690 27546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:16.069900 27546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:16.071041 27546 hybrid_clock.cc:648] HybridClock initialized: now 1790303716071030 us; error 27 us; skew 500 ppm
Sep 25 02:35:16 dist-test-slave-q8mg krb5kdc[27305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303716, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:16.073621 27546 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:35:16.074115 27546 webserver.cc:460] Webserver started at http://127.10.13.2:34407/ using document root <none> and password file <none>
I20260925 02:35:16.074376 27546 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:16.074424 27546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:16.074618 27546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:16.075693 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "bcbf5d87655244b4bba28ed513d97a01"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "2b1e2fe6bd3296443e6084b357580a89"
server_key_iv: "98c7b21cae9d504bd58cbdd01b852d09"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.076057 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "bcbf5d87655244b4bba28ed513d97a01"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "2b1e2fe6bd3296443e6084b357580a89"
server_key_iv: "98c7b21cae9d504bd58cbdd01b852d09"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.077512 27546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:16.078243 27561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.078418 27546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:16.078538 27546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "bcbf5d87655244b4bba28ed513d97a01"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "2b1e2fe6bd3296443e6084b357580a89"
server_key_iv: "98c7b21cae9d504bd58cbdd01b852d09"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.078616 27546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:16.085777 27546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:16.087598 27546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:16.087772 27546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:16.088008 27546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:16.088325 27546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:16.088378 27546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.088418 27546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:16.088474 27546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.094631 27546 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:35223
I20260925 02:35:16.094678 27674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:35223 every 8 connection(s)
I20260925 02:35:16.095006 27546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:35:16 dist-test-slave-q8mg krb5kdc[27305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303716, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:16.102695 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 27546
I20260925 02:35:16.102814 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:35:16.103039 10292 external_mini_cluster.cc:1428] Setting key 013405cc9718bc6e144aae997d7220a3
I20260925 02:35:16.104903 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.095785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:38109 (local address 127.10.13.62:43605)
0925 02:35:16.095875 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.095878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.096044 (+ 166us) server_negotiation.cc:407] Connection header received
0925 02:35:16.096534 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.096535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.096545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.096594 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:16.097543 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.098037 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.099193 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.099378 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.099438 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.101697 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:16.101699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:16.101699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:16.101706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:16.103296 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:16.103731 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:16.103732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:16.103733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:16.103815 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:16.104120 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:16.104121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:16.104122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:16.104270 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:16.104345 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.104639 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.104735 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:35:16.105027 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.095726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43605 (local address 127.10.13.2:38109)
0925 02:35:16.095923 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.095961 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.096452 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.096654 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.096658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.097259 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:16.097466 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.097470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.098125 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.098126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.099111 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.099114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.099453 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.099454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.099552 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.100115 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:16.100124 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:16.101601 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:16.103384 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:16.103392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:16.103395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:16.103637 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:16.103909 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:16.103910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:16.103911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:16.104029 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:16.104365 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:16.104372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:16.104501 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.104595 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.104725 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260925 02:35:16.105880 27675 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43605
I20260925 02:35:16.105995 27675 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:16.106204 27675 heartbeater.cc:511] Master 127.10.13.62:43605 requested a full tablet report, sending...
I20260925 02:35:16.106627 27350 ts_manager.cc:194] Registered new tserver with Master: bcbf5d87655244b4bba28ed513d97a01 (127.10.13.2:35223)
I20260925 02:35:16.107161 27350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:38109
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:35:16.129555 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43605
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:16.215430 27680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:16.215651 27680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:16.215675 27680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:16.216778 27680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:16.216814 27680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:16.216840 27680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:35:16.218432 27680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:16.218627 27680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:16.218808 27680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:16.223949 27688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.223978 27685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.223970 27686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.224004 27680 server_base.cc:1034] running on GCE node
I20260925 02:35:16.224406 27680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:16.224659 27680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:16.225804 27680 hybrid_clock.cc:648] HybridClock initialized: now 1790303716225770 us; error 49 us; skew 500 ppm
Sep 25 02:35:16 dist-test-slave-q8mg krb5kdc[27305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303716, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:16.228324 27680 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:35:16.228806 27680 webserver.cc:460] Webserver started at http://127.10.13.3:42779/ using document root <none> and password file <none>
I20260925 02:35:16.229069 27680 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:16.229143 27680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:16.229300 27680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:16.230459 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "3eef3f86b86048d6b8e52416a7c9a294"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "86fa28ca1df2f41bfd2678f6cc2782bd"
server_key_iv: "06d405bc77e06a6d97e0cabd33e5411f"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.230820 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "3eef3f86b86048d6b8e52416a7c9a294"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "86fa28ca1df2f41bfd2678f6cc2782bd"
server_key_iv: "06d405bc77e06a6d97e0cabd33e5411f"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.232247 27680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:16.233011 27695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.233182 27680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:16.233286 27680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "3eef3f86b86048d6b8e52416a7c9a294"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "86fa28ca1df2f41bfd2678f6cc2782bd"
server_key_iv: "06d405bc77e06a6d97e0cabd33e5411f"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.233376 27680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:16.252108 27680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:16.254631 27680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:16.254809 27680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:16.255069 27680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:16.255411 27680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:16.255468 27680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.255514 27680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:16.255539 27680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.262014 27680 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:45505
I20260925 02:35:16.262097 27808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:45505 every 8 connection(s)
I20260925 02:35:16.262467 27680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:35:16.265275 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 27680
I20260925 02:35:16.265386 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:35:16.265622 10292 external_mini_cluster.cc:1428] Setting key acd002e037d8de31d70c52dce60da897
Sep 25 02:35:16 dist-test-slave-q8mg krb5kdc[27305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303716, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:16.272498 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.263312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:49899 (local address 127.10.13.62:43605)
0925 02:35:16.263379 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.263383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.263599 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:35:16.264150 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.264152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.264165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.264225 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:16.265562 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.266225 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.267395 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.267580 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.267640 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.269954 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:16.269956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:16.269956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:16.269964 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:16.271216 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:16.271601 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:16.271602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:16.271602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:16.271653 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:16.271913 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:16.271914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:16.271914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:16.272025 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:16.272080 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.272355 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.272429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:35:16.272648 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.263306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43605 (local address 127.10.13.3:49899)
0925 02:35:16.263473 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.263511 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.264041 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.264303 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.264307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.264996 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:16.265344 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.265353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.266326 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.266327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.267310 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.267312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.267658 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.267659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.267768 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.268232 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:16.268241 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:16.269863 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:16.271292 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:16.271299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:16.271301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:16.271534 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:16.271751 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:16.271751 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:16.271752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:16.271854 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:16.272085 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:16.272090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:16.272211 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.272295 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.272422 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260925 02:35:16.273178 27809 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43605
I20260925 02:35:16.273294 27809 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:16.273502 27809 heartbeater.cc:511] Master 127.10.13.62:43605 requested a full tablet report, sending...
I20260925 02:35:16.273904 27350 ts_manager.cc:194] Registered new tserver with Master: 3eef3f86b86048d6b8e52416a7c9a294 (127.10.13.3:45505)
I20260925 02:35:16.274247 27350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:49899
I20260925 02:35:16.277532 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:35:16.284773 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.278888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.10.13.62:43605)
0925 02:35:16.278945 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.278947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.279023 (+ 76us) server_negotiation.cc:407] Connection header received
0925 02:35:16.279061 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.279062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.279070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.279118 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:16.279614 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.280072 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.280800 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.281049 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.281104 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.282439 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:16.282440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:16.282441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:16.282447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:16.283817 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:16.284147 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:16.284148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:16.284148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:16.284197 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:16.284340 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:16.284341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:16.284341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:16.284477 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:16.284553 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.284649 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.284707 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:35:16.286396 27350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:35:16.288033 27350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:35:16.294624 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.290274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:35223 (local address 127.0.0.1:46936)
0925 02:35:16.290394 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.290440 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.290521 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.290930 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.290950 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.290952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:16.291025 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.291027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.292154 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.292155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.293365 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.293367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.294290 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.294292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.294383 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.294399 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.294481 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.294539 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260925 02:35:16.294637 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.290562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.10.13.2:35223)
0925 02:35:16.290738 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.290743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.290755 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:35:16.290799 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.290801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.290818 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.290911 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:16.291111 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.292051 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.293455 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.294207 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.294277 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.294337 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.294508 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.294552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260925 02:35:16.294950 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.290275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:33763 (local address 127.0.0.1:36654)
0925 02:35:16.291155 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.291164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.291205 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.291484 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.291485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.291487 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:16.291537 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.291539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.292782 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.292784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.293795 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.293797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.294605 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.294606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.294668 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.294673 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.294820 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.294872 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":43,"thread_start_us":67,"threads_started":1}
I20260925 02:35:16.295010 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.290790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.10.13.1:33763)
0925 02:35:16.290985 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.290989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.291265 (+ 276us) server_negotiation.cc:407] Connection header received
0925 02:35:16.291310 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.291313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.291330 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.291398 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:16.291616 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.292685 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.293887 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.294456 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.294517 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.294572 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.294868 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.294914 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260925 02:35:16.295363 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.290275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:45505 (local address 127.0.0.1:37428)
0925 02:35:16.290983 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.290994 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.291048 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.291395 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.291396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.291398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:16.291437 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.291438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.292759 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.292761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.294113 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.294116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.295120 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.295122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.295186 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.295189 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.295244 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.295281 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":38,"spinlock_wait_cycles":768,"thread_start_us":75,"threads_started":1}
I20260925 02:35:16.295670 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.290533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.10.13.3:45505)
0925 02:35:16.290719 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.290724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.291044 (+ 320us) server_negotiation.cc:407] Connection header received
0925 02:35:16.291208 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.291211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.291230 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.291312 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:16.291518 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.292626 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.294218 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.294953 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.295024 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.295079 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.295260 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.295584 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260925 02:35:16.295779 27475 tablet_service.cc:1467] Processing CreateTablet for tablet 64c50bb637a24508a3206304d7691062 (DEFAULT_TABLE table=test-table [id=a28fe66a138b46818942ef9d5cb459a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:16.295782 27609 tablet_service.cc:1467] Processing CreateTablet for tablet 64c50bb637a24508a3206304d7691062 (DEFAULT_TABLE table=test-table [id=a28fe66a138b46818942ef9d5cb459a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:16.296154 27609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c50bb637a24508a3206304d7691062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:16.296154 27475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c50bb637a24508a3206304d7691062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:16.296265 27743 tablet_service.cc:1467] Processing CreateTablet for tablet 64c50bb637a24508a3206304d7691062 (DEFAULT_TABLE table=test-table [id=a28fe66a138b46818942ef9d5cb459a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:16.296525 27743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c50bb637a24508a3206304d7691062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:16.299557 27825 tablet_bootstrap.cc:492] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7: Bootstrap starting.
I20260925 02:35:16.299782 27823 tablet_bootstrap.cc:492] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: Bootstrap starting.
I20260925 02:35:16.300211 27825 tablet_bootstrap.cc:654] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:16.300434 27823 tablet_bootstrap.cc:654] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:16.300791 27825 log.cc:824] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:16.300961 27823 log.cc:824] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:16.301221 27824 tablet_bootstrap.cc:492] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Bootstrap starting.
I20260925 02:35:16.301640 27825 tablet_bootstrap.cc:492] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7: No bootstrap required, opened a new log
I20260925 02:35:16.301730 27825 ts_tablet_manager.cc:1397] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:35:16.301739 27823 tablet_bootstrap.cc:492] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: No bootstrap required, opened a new log
I20260925 02:35:16.301839 27823 ts_tablet_manager.cc:1397] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:35:16.303467 27825 raft_consensus.cc:348] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.303514 27823 raft_consensus.cc:348] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.303635 27823 raft_consensus.cc:374] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:16.303635 27825 raft_consensus.cc:374] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:16.303679 27823 raft_consensus.cc:729] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eef3f86b86048d6b8e52416a7c9a294, State: Initialized, Role: FOLLOWER
I20260925 02:35:16.303683 27825 raft_consensus.cc:729] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74676206e6454f6e9fc43a082b207ee7, State: Initialized, Role: FOLLOWER
I20260925 02:35:16.303807 27823 consensus_queue.cc:260] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.303807 27825 consensus_queue.cc:260] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.304100 27823 ts_tablet_manager.cc:1428] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 02:35:16.304113 27541 heartbeater.cc:503] Master 127.10.13.62:43605 was elected leader, sending a full tablet report...
I20260925 02:35:16.304113 27809 heartbeater.cc:503] Master 127.10.13.62:43605 was elected leader, sending a full tablet report...
I20260925 02:35:16.304100 27825 ts_tablet_manager.cc:1428] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:35:16.305048 27824 tablet_bootstrap.cc:654] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:16.305630 27824 log.cc:824] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:16.306362 27824 tablet_bootstrap.cc:492] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: No bootstrap required, opened a new log
I20260925 02:35:16.306419 27824 ts_tablet_manager.cc:1397] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 02:35:16.308070 27824 raft_consensus.cc:348] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.308209 27824 raft_consensus.cc:374] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:16.308233 27824 raft_consensus.cc:729] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbf5d87655244b4bba28ed513d97a01, State: Initialized, Role: FOLLOWER
I20260925 02:35:16.308390 27824 consensus_queue.cc:260] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.308658 27824 ts_tablet_manager.cc:1428] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:35:16.308710 27675 heartbeater.cc:503] Master 127.10.13.62:43605 was elected leader, sending a full tablet report...
W20260925 02:35:16.345587 27676 tablet.cc:2369] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:35:16.373431 27829 raft_consensus.cc:482] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:16.373575 27829 raft_consensus.cc:504] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.373911 27829 leader_election.cc:288] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74676206e6454f6e9fc43a082b207ee7 (127.10.13.1:33763), bcbf5d87655244b4bba28ed513d97a01 (127.10.13.2:35223)
I20260925 02:35:16.378114 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.374234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:36197 (local address 127.10.13.1:33763)
0925 02:35:16.374280 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.374281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.374573 (+ 292us) server_negotiation.cc:407] Connection header received
0925 02:35:16.374603 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.374604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.374634 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.374694 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:16.374838 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.375525 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.377075 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.377744 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.377806 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.377817 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.377998 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.378039 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:35:16.378114 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.374241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:46805 (local address 127.10.13.2:35223)
0925 02:35:16.374301 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.374303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.374314 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:35:16.374348 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.374349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.374401 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.374476 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:16.374667 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.375483 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.377074 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.377710 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.377794 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.377807 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.377998 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.378040 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:35:16.378137 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.374130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:35223 (local address 127.10.13.3:46805)
0925 02:35:16.374200 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.374217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.374284 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.374493 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.374494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.374496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:16.374580 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.374582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.375587 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.375588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.376966 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.376969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.377807 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.377808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.377908 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.377910 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.377973 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.378019 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":90}
I20260925 02:35:16.378149 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.374130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:33763 (local address 127.10.13.3:36197)
0925 02:35:16.374491 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.374501 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.374558 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.374713 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.374714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.374715 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:16.374767 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.374768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.375602 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.375602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.376965 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.376969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.377812 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.377812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.377905 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.377910 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.377978 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.378017 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":157,"thread_start_us":182,"threads_started":1}
I20260925 02:35:16.378572 27629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c50bb637a24508a3206304d7691062" candidate_uuid: "3eef3f86b86048d6b8e52416a7c9a294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbf5d87655244b4bba28ed513d97a01" is_pre_election: true
I20260925 02:35:16.378790 27629 raft_consensus.cc:2454] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eef3f86b86048d6b8e52416a7c9a294 in term 0.
I20260925 02:35:16.378782 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c50bb637a24508a3206304d7691062" candidate_uuid: "3eef3f86b86048d6b8e52416a7c9a294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74676206e6454f6e9fc43a082b207ee7" is_pre_election: true
I20260925 02:35:16.378914 27495 raft_consensus.cc:2454] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eef3f86b86048d6b8e52416a7c9a294 in term 0.
I20260925 02:35:16.379062 27696 leader_election.cc:302] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eef3f86b86048d6b8e52416a7c9a294, bcbf5d87655244b4bba28ed513d97a01; no voters:
I20260925 02:35:16.379222 27829 raft_consensus.cc:2781] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:35:16.379319 27829 raft_consensus.cc:482] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:35:16.379365 27829 raft_consensus.cc:3037] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:16.380124 27829 raft_consensus.cc:504] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.380292 27829 leader_election.cc:288] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [CANDIDATE]: Term 1 election: Requested vote from peers 74676206e6454f6e9fc43a082b207ee7 (127.10.13.1:33763), bcbf5d87655244b4bba28ed513d97a01 (127.10.13.2:35223)
I20260925 02:35:16.380427 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c50bb637a24508a3206304d7691062" candidate_uuid: "3eef3f86b86048d6b8e52416a7c9a294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74676206e6454f6e9fc43a082b207ee7"
I20260925 02:35:16.380501 27629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c50bb637a24508a3206304d7691062" candidate_uuid: "3eef3f86b86048d6b8e52416a7c9a294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbf5d87655244b4bba28ed513d97a01"
I20260925 02:35:16.380517 27495 raft_consensus.cc:3037] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:16.380573 27629 raft_consensus.cc:3037] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:16.381512 27629 raft_consensus.cc:2454] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eef3f86b86048d6b8e52416a7c9a294 in term 1.
I20260925 02:35:16.381512 27495 raft_consensus.cc:2454] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eef3f86b86048d6b8e52416a7c9a294 in term 1.
I20260925 02:35:16.381680 27696 leader_election.cc:302] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eef3f86b86048d6b8e52416a7c9a294, bcbf5d87655244b4bba28ed513d97a01; no voters:
I20260925 02:35:16.381825 27829 raft_consensus.cc:2781] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:16.382000 27829 raft_consensus.cc:686] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 1 LEADER]: Becoming Leader. State: Replica: 3eef3f86b86048d6b8e52416a7c9a294, State: Running, Role: LEADER
I20260925 02:35:16.382082 27829 consensus_queue.cc:237] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } }
I20260925 02:35:16.382969 27349 catalog_manager.cc:5495] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eef3f86b86048d6b8e52416a7c9a294 (127.10.13.3). New cstate: current_term: 1 leader_uuid: "3eef3f86b86048d6b8e52416a7c9a294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eef3f86b86048d6b8e52416a7c9a294" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 45505 } health_report { overall_health: HEALTHY } } }
I20260925 02:35:16.411731 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.408436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.10.13.3:45505)
0925 02:35:16.408507 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.408508 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.408519 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:35:16.408589 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.408590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.408614 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.408707 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:16.409084 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.409863 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.410694 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.410958 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.411022 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.411251 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:16.411426 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:16.411514 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.411561 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.411596 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:35:16.413897 27495 raft_consensus.cc:1261] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 1 FOLLOWER]: Refusing update from remote peer 3eef3f86b86048d6b8e52416a7c9a294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:16.413897 27629 raft_consensus.cc:1261] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 1 FOLLOWER]: Refusing update from remote peer 3eef3f86b86048d6b8e52416a7c9a294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:16.414325 27833 consensus_queue.cc:1035] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74676206e6454f6e9fc43a082b207ee7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:16.414474 27829 consensus_queue.cc:1035] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcbf5d87655244b4bba28ed513d97a01" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:16.417011 27836 mvcc.cc:205] Tried to move back new op lower bound from 7333084022427787264 to 7333084022301650944. Current Snapshot: MvccSnapshot[applied={T|T < 7333084022427787264}]
I20260925 02:35:16.419958 27349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:16.420068 27349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:16.420969 27349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25cf8a789db047d7aa7c748b36f5a265: Sending DeleteTablet for 3 replicas of tablet 64c50bb637a24508a3206304d7691062
I20260925 02:35:16.421314 27475 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c50bb637a24508a3206304d7691062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:16 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:36654
I20260925 02:35:16.421360 27609 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c50bb637a24508a3206304d7691062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:16 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:46936
I20260925 02:35:16.421360 27743 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c50bb637a24508a3206304d7691062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:16 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:37428
I20260925 02:35:16.421598 27846 tablet_replica.cc:330] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7: stopping tablet replica
I20260925 02:35:16.421712 27847 tablet_replica.cc:330] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: stopping tablet replica
I20260925 02:35:16.421712 27846 raft_consensus.cc:2229] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:16.421824 27846 raft_consensus.cc:2258] T 64c50bb637a24508a3206304d7691062 P 74676206e6454f6e9fc43a082b207ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:16.421904 27848 tablet_replica.cc:330] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: stopping tablet replica
I20260925 02:35:16.421960 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 27412
I20260925 02:35:16.425002 27848 raft_consensus.cc:2229] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:35:16.425158 27848 raft_consensus.cc:2258] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:16.425284 27847 raft_consensus.cc:2229] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:16.425472 27847 raft_consensus.cc:2258] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:16.425626 27848 ts_tablet_manager.cc:1905] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:16.425956 27847 ts_tablet_manager.cc:1905] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:16.427456 27848 ts_tablet_manager.cc:1918] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:16.427599 27848 log.cc:1196] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/64c50bb637a24508a3206304d7691062
I20260925 02:35:16.427973 27848 ts_tablet_manager.cc:1939] T 64c50bb637a24508a3206304d7691062 P 3eef3f86b86048d6b8e52416a7c9a294: Deleting consensus metadata
I20260925 02:35:16.427969 27847 ts_tablet_manager.cc:1918] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:16.428059 27847 log.cc:1196] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/64c50bb637a24508a3206304d7691062
I20260925 02:35:16.428385 27847 ts_tablet_manager.cc:1939] T 64c50bb637a24508a3206304d7691062 P bcbf5d87655244b4bba28ed513d97a01: Deleting consensus metadata
W20260925 02:35:16.428581 27337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:33763 (error 108)
I20260925 02:35:16.428859 27336 catalog_manager.cc:4841] TS 3eef3f86b86048d6b8e52416a7c9a294 (127.10.13.3:45505): tablet 64c50bb637a24508a3206304d7691062 (table test-table [id=a28fe66a138b46818942ef9d5cb459a8]) successfully deleted
I20260925 02:35:16.429109 27334 catalog_manager.cc:4841] TS bcbf5d87655244b4bba28ed513d97a01 (127.10.13.2:35223): tablet 64c50bb637a24508a3206304d7691062 (table test-table [id=a28fe66a138b46818942ef9d5cb459a8]) successfully deleted
W20260925 02:35:16.429311 27819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:16.429148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:33763 (local address 127.0.0.1:36670)
0925 02:35:16.429217 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.429265 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:33763: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:35:16.429445 27337 catalog_manager.cc:4584] TS 74676206e6454f6e9fc43a082b207ee7 (127.10.13.1:33763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64c50bb637a24508a3206304d7691062: Network error: Client connection negotiation failed: client connection to 127.10.13.1:33763: connect: Connection reset by peer (error 104)
I20260925 02:35:16.429528 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 27546
I20260925 02:35:16.436560 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 27680
I20260925 02:35:16.441606 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 27318
2026-09-25T02:35:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1989 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T02:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:16Z Disabled control of system clock
I20260925 02:35:16.470012 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34899
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:34899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:16.553220 27853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:16.553452 27853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:16.553472 27853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:16.554682 27853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:16.554728 27853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:16.554741 27853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:16.554754 27853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:16.554766 27853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:16.556052 27853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:34899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:16.556259 27853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:16.556523 27853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:16.561486 27858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.561544 27861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.561702 27853 server_base.cc:1034] running on GCE node
W20260925 02:35:16.561535 27859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.562001 27853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:16.562258 27853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:16.563416 27853 hybrid_clock.cc:648] HybridClock initialized: now 1790303716563380 us; error 49 us; skew 500 ppm
I20260925 02:35:16.565094 27853 webserver.cc:460] Webserver started at http://127.10.13.62:38989/ using document root <none> and password file <none>
I20260925 02:35:16.565397 27853 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:16.565469 27853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:16.565608 27853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:16.566700 27853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "a782c431952bab31efbf8551e2677c6f"
server_key_iv: "1e38218d5d9090336f13806b7658add6"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.567085 27853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "a782c431952bab31efbf8551e2677c6f"
server_key_iv: "1e38218d5d9090336f13806b7658add6"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.568480 27853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:16.569433 27867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.569646 27853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:35:16.569761 27853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "a782c431952bab31efbf8551e2677c6f"
server_key_iv: "1e38218d5d9090336f13806b7658add6"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.569854 27853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:16.583556 27853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:16.583905 27853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:16.584069 27853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:16.588200 27853 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:34899
I20260925 02:35:16.588236 27919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:34899 every 8 connection(s)
I20260925 02:35:16.588599 27853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:16.589658 27920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:16.591991 27920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26: Bootstrap starting.
I20260925 02:35:16.592589 27920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:16.592914 27920 log.cc:824] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:16.593639 27920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26: No bootstrap required, opened a new log
I20260925 02:35:16.594272 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 27853
I20260925 02:35:16.594419 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:16.594623 10292 external_mini_cluster.cc:1428] Setting key 8da8ee1bbf01811bc595af7bc84d5645
I20260925 02:35:16.595851 27920 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34899 } }
I20260925 02:35:16.596040 27920 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:16.596081 27920 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3fd1e4d67fa4d4d9f25a6cde5eaca26, State: Initialized, Role: FOLLOWER
I20260925 02:35:16.596243 27920 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34899 } }
I20260925 02:35:16.596334 27920 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:16.596366 27920 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:16.596408 27920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:16.597195 27920 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34899 } }
I20260925 02:35:16.597347 27920 leader_election.cc:302] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3fd1e4d67fa4d4d9f25a6cde5eaca26; no voters:
I20260925 02:35:16.597512 27920 leader_election.cc:288] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:16.597604 27925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:16.597754 27920 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:16.597787 27925 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [term 1 LEADER]: Becoming Leader. State: Replica: d3fd1e4d67fa4d4d9f25a6cde5eaca26, State: Running, Role: LEADER
I20260925 02:35:16.597998 27925 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34899 } }
I20260925 02:35:16.598431 27926 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34899 } } }
I20260925 02:35:16.598511 27926 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:16.598750 27927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3fd1e4d67fa4d4d9f25a6cde5eaca26. Latest consensus state: current_term: 1 leader_uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fd1e4d67fa4d4d9f25a6cde5eaca26" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34899 } } }
I20260925 02:35:16.598879 27927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:16.599037 27934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:16.599576 27934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:16.601565 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.595209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.10.13.62:34899)
0925 02:35:16.595477 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.595487 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.595527 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:35:16.595805 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.595814 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.595897 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.596054 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:16.596305 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.598161 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.599307 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.599909 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.600038 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.600778 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:16.600783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:16.600783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:16.601024 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:16.601068 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:16.601080 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:16.601162 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.601246 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.601341 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
I20260925 02:35:16.602689 27934 catalog_manager.cc:1347] Generated new cluster ID: 2b32dcb7a6b840999b1351915f745083
I20260925 02:35:16.602743 27934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:16.609445 27934 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:16.609977 27934 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:16.627745 27934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26: Generated new TSK 0
I20260925 02:35:16.628021 27934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:16.631873 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34899
--builtin_ntp_servers=127.10.13.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:16.716805 27944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:16.717073 27944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:16.717098 27944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:16.718318 27944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:16.718369 27944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:16.718396 27944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:35:16.719784 27944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:16.720048 27944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:16.720383 27944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:16.726725 27950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.726785 27944 server_base.cc:1034] running on GCE node
W20260925 02:35:16.726740 27952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.726733 27949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.727185 27944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:16.727419 27944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:16.728574 27944 hybrid_clock.cc:648] HybridClock initialized: now 1790303716728562 us; error 29 us; skew 500 ppm
I20260925 02:35:16.729955 27944 webserver.cc:460] Webserver started at http://127.10.13.1:39295/ using document root <none> and password file <none>
I20260925 02:35:16.730230 27944 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:16.730300 27944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:16.730453 27944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:16.731567 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "ded31e45ba31495b8ec0b498d54769a7"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "97c4fcdb5d7b1a9a2f07433e29db601b"
server_key_iv: "b8327d22223fba344920998195727dc3"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.731940 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "ded31e45ba31495b8ec0b498d54769a7"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "97c4fcdb5d7b1a9a2f07433e29db601b"
server_key_iv: "b8327d22223fba344920998195727dc3"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.733371 27944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 02:35:16.734172 27958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.734344 27944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:16.734440 27944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "ded31e45ba31495b8ec0b498d54769a7"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "97c4fcdb5d7b1a9a2f07433e29db601b"
server_key_iv: "b8327d22223fba344920998195727dc3"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.734522 27944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:16.750510 27944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:16.750862 27944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:16.751041 27944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:16.751288 27944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:16.751636 27944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:16.751689 27944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.751730 27944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:16.751757 27944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.757625 27944 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:34693
I20260925 02:35:16.757709 28071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:34693 every 8 connection(s)
I20260925 02:35:16.758087 27944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:35:16.763422 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.758936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:59173 (local address 127.10.13.62:34899)
0925 02:35:16.759009 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.759012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.759267 (+ 255us) server_negotiation.cc:407] Connection header received
0925 02:35:16.759762 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.759763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.759776 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.759852 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:16.760495 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.761086 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.762072 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.762259 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.762310 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.763120 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:16.763122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:16.763122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:16.763130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:16.763148 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:16.763157 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:16.763219 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.763304 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.763336 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:35:16.763652 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.758911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34899 (local address 127.10.13.1:59173)
0925 02:35:16.759136 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.759173 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.759676 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.759892 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.759895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.759904 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:16.760417 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.760422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.761168 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.761169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.762002 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.762004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.762334 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.762335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.762453 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.762965 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:16.762965 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:16.762995 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:16.763018 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:16.763219 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:16.763228 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:16.763231 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.763306 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.763424 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260925 02:35:16.764173 28072 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34899
I20260925 02:35:16.764277 28072 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:16.764472 28072 heartbeater.cc:511] Master 127.10.13.62:34899 requested a full tablet report, sending...
I20260925 02:35:16.765097 27884 ts_manager.cc:194] Registered new tserver with Master: ded31e45ba31495b8ec0b498d54769a7 (127.10.13.1:34693)
I20260925 02:35:16.765710 27884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.1:59173
I20260925 02:35:16.767318 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 27944
I20260925 02:35:16.767417 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:35:16.767568 10292 external_mini_cluster.cc:1428] Setting key bdeed6f1775130b0052d691403f14a31
I20260925 02:35:16.768453 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34899
--builtin_ntp_servers=127.10.13.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:16.851759 28075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:16.851989 28075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:16.852015 28075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:16.853230 28075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:16.853273 28075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:16.853298 28075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:35:16.854640 28075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:16.854868 28075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:16.855134 28075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:16.860292 28083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.860432 28075 server_base.cc:1034] running on GCE node
W20260925 02:35:16.860275 28081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.860275 28080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.860726 28075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:16.861011 28075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:16.862170 28075 hybrid_clock.cc:648] HybridClock initialized: now 1790303716862156 us; error 27 us; skew 500 ppm
I20260925 02:35:16.863538 28075 webserver.cc:460] Webserver started at http://127.10.13.2:40741/ using document root <none> and password file <none>
I20260925 02:35:16.863801 28075 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:16.863852 28075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:16.864033 28075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:16.865221 28075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "c5393eff53fe4110b7b33a18004f7950"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "cc25cc611ccb3b82406b0d04a76278f1"
server_key_iv: "91fa8d152dd6347edba158125aa89111"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.865594 28075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "c5393eff53fe4110b7b33a18004f7950"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "cc25cc611ccb3b82406b0d04a76278f1"
server_key_iv: "91fa8d152dd6347edba158125aa89111"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.866954 28075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:35:16.867700 28089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.867890 28075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:16.867983 28075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "c5393eff53fe4110b7b33a18004f7950"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "cc25cc611ccb3b82406b0d04a76278f1"
server_key_iv: "91fa8d152dd6347edba158125aa89111"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.868060 28075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:16.877919 28075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:16.878237 28075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:16.878389 28075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:16.878623 28075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:16.878958 28075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:16.878993 28075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.879015 28075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:16.879060 28075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.885030 28075 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:39933
I20260925 02:35:16.885087 28202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:39933 every 8 connection(s)
I20260925 02:35:16.885438 28075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:35:16.890643 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.886213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:58373 (local address 127.10.13.62:34899)
0925 02:35:16.886282 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:16.886285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:16.886556 (+ 271us) server_negotiation.cc:407] Connection header received
0925 02:35:16.887051 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:16.887052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:16.887062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:16.887130 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:16.887791 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.888285 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.889341 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.889526 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.889582 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.890441 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:16.890442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:16.890442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:16.890447 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:16.890458 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:16.890461 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:16.890494 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:16.890562 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 02:35:16.890584 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:35:16.890851 28205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:16.886214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34899 (local address 127.10.13.2:58373)
0925 02:35:16.886425 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:16.886469 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:16.886964 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:16.887188 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:16.887191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:16.887199 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:16.887719 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.887723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.888379 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.888380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.889262 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:16.889265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:16.889604 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:16.889604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:16.889717 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:16.890302 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:16.890302 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:16.890343 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:16.890366 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:16.890502 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:16.890505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:16.890507 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:35:16.890553 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 02:35:16.890631 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260925 02:35:16.891371 28203 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34899
I20260925 02:35:16.891494 28203 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:16.891683 28203 heartbeater.cc:511] Master 127.10.13.62:34899 requested a full tablet report, sending...
I20260925 02:35:16.892082 27884 ts_manager.cc:194] Registered new tserver with Master: c5393eff53fe4110b7b33a18004f7950 (127.10.13.2:39933)
I20260925 02:35:16.892467 27884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.2:58373
I20260925 02:35:16.893532 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 28075
I20260925 02:35:16.893625 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:35:16.893800 10292 external_mini_cluster.cc:1428] Setting key e60fe64b36e111a86a41272e8d4852db
I20260925 02:35:16.894685 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34899
--builtin_ntp_servers=127.10.13.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:16.977025 28206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:16.977257 28206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:16.977294 28206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:16.978452 28206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:16.978518 28206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:16.978560 28206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:35:16.979918 28206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:16.980206 28206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:16.980518 28206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:35:16.985612 28206 server_base.cc:1034] running on GCE node
W20260925 02:35:16.985577 28214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.985603 28211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:16.985577 28212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:16.985962 28206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:16.986210 28206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:16.987378 28206 hybrid_clock.cc:648] HybridClock initialized: now 1790303716987362 us; error 27 us; skew 500 ppm
I20260925 02:35:16.988714 28206 webserver.cc:460] Webserver started at http://127.10.13.3:43179/ using document root <none> and password file <none>
I20260925 02:35:16.989053 28206 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:16.989130 28206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:16.989288 28206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:16.990370 28206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "6b585776762446a3ba5cd9fabdb3fd12"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "e808be41d7027ca4219eb97f855d8356"
server_key_iv: "781afc9182a290e7c1eab5b9ffb81eea"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.990732 28206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "6b585776762446a3ba5cd9fabdb3fd12"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "e808be41d7027ca4219eb97f855d8356"
server_key_iv: "781afc9182a290e7c1eab5b9ffb81eea"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.992026 28206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:16.992760 28220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:16.992967 28206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:16.993072 28206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "6b585776762446a3ba5cd9fabdb3fd12"
format_stamp: "Formatted at 2026-09-25 02:35:16 on dist-test-slave-q8mg"
server_key: "e808be41d7027ca4219eb97f855d8356"
server_key_iv: "781afc9182a290e7c1eab5b9ffb81eea"
server_key_version: "encryptionkey@0"
I20260925 02:35:16.993160 28206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:17.016446 28206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:17.016799 28206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:17.016986 28206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:17.017237 28206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:17.017601 28206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:17.017655 28206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:17.017701 28206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:17.017725 28206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:17.023630 28206 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:46719
I20260925 02:35:17.023687 28333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:46719 every 8 connection(s)
I20260925 02:35:17.024056 28206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:35:17.029145 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.024792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:59215 (local address 127.10.13.62:34899)
0925 02:35:17.024899 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:17.024901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:17.025130 (+ 229us) server_negotiation.cc:407] Connection header received
0925 02:35:17.025599 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:17.025600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:17.025610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:17.025663 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:17.026331 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.026818 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.027769 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.027959 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.028009 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.028864 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:17.028865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:17.028865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:17.028910 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:17.028923 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:17.028935 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:17.028990 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:17.029060 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:35:17.029085 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:35:17.029395 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.024777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:34899 (local address 127.10.13.3:59215)
0925 02:35:17.025009 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:17.025045 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:17.025517 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:17.025717 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:17.025721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:17.025729 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:17.026252 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.026257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.026901 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.026902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.027693 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.027695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.028033 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.028034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.028142 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.028734 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:17.028735 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:17.028765 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:17.028789 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:17.028994 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:17.028997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:17.028999 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:35:17.029051 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:35:17.029132 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":126,"threads_started":1}
I20260925 02:35:17.029633 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 28206
I20260925 02:35:17.029747 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:35:17.029944 10292 external_mini_cluster.cc:1428] Setting key c222946bfd28568e0bb49355af77a97c
I20260925 02:35:17.030001 28334 heartbeater.cc:347] Connected to a master server at 127.10.13.62:34899
I20260925 02:35:17.030090 28334 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:17.030313 28334 heartbeater.cc:511] Master 127.10.13.62:34899 requested a full tablet report, sending...
I20260925 02:35:17.030728 27883 ts_manager.cc:194] Registered new tserver with Master: 6b585776762446a3ba5cd9fabdb3fd12 (127.10.13.3:46719)
I20260925 02:35:17.031123 27883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.3:59215
I20260925 02:35:17.041982 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:35:17.046638 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.043437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.10.13.62:34899)
0925 02:35:17.043513 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:17.043516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:17.043591 (+ 75us) server_negotiation.cc:407] Connection header received
0925 02:35:17.043698 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:17.043699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:17.043714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:17.043786 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:17.043986 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.044482 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.045358 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.045555 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.045629 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.046372 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:17.046373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:17.046373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:17.046378 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:17.046398 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:17.046402 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:17.046463 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:17.046518 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 02:35:17.046544 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:35:17.048487 27883 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:35:17.050263 27883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:35:17.057199 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.053009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.10.13.1:34693)
0925 02:35:17.053225 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:17.053231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:17.053243 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:35:17.053305 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:17.053307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:17.053325 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:17.053411 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:17.053600 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.054778 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.056024 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.056604 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.056677 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.056810 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:17.057060 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 02:35:17.057106 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260925 02:35:17.057211 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.053045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:39933 (local address 127.0.0.1:56000)
0925 02:35:17.053216 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:17.053225 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:17.053262 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:17.053802 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:17.053802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:17.053805 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:17.053840 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.053841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.055034 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.055035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.055951 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.055954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.056824 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.056825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.056928 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.056947 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:35:17.057025 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:35:17.057094 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260925 02:35:17.057211 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.052858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:34693 (local address 127.0.0.1:39408)
0925 02:35:17.053037 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:17.053049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:17.053106 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:17.053427 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:17.053431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:17.053434 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:17.053508 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.053510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.054880 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.054881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.055963 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.055965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.056840 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.056841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.056928 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.056947 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:35:17.057026 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 02:35:17.057094 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":54,"thread_start_us":100,"threads_started":1}
I20260925 02:35:17.057284 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.052718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.10.13.3:46719)
0925 02:35:17.053384 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:17.053389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:17.053398 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:17.053435 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:17.053436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:17.053450 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:17.053522 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:17.053653 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.054460 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.055961 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.056703 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.056779 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.056892 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:17.057150 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 02:35:17.057191 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":204,"threads_started":1}
I20260925 02:35:17.057489 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.052599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:46719 (local address 127.0.0.1:51614)
0925 02:35:17.052743 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:17.052784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:17.052904 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:17.053524 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:17.053525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:17.053527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:17.053576 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.053578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.054555 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.054556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.055861 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.055873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.056931 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.056932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.056999 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.057003 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:35:17.057059 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:35:17.057107 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":8,"thread_start_us":97,"threads_started":1}
I20260925 02:35:17.058024 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.053154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.10.13.2:39933)
0925 02:35:17.053339 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:17.053343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:17.053628 (+ 285us) server_negotiation.cc:407] Connection header received
0925 02:35:17.053675 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:17.053677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:17.053693 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:17.053786 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:17.053913 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.054953 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.056049 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.056728 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.056795 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.056907 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:17.057922 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0925 02:35:17.057961 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260925 02:35:17.058243 28268 tablet_service.cc:1467] Processing CreateTablet for tablet d802dd1b82e84bba9e99d9139074afbf (DEFAULT_TABLE table=test-table [id=c8cd20f7b590464f8f2b92eb85c62795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:17.058655 28268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d802dd1b82e84bba9e99d9139074afbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:17.058737 28137 tablet_service.cc:1467] Processing CreateTablet for tablet d802dd1b82e84bba9e99d9139074afbf (DEFAULT_TABLE table=test-table [id=c8cd20f7b590464f8f2b92eb85c62795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:17.058768 28006 tablet_service.cc:1467] Processing CreateTablet for tablet d802dd1b82e84bba9e99d9139074afbf (DEFAULT_TABLE table=test-table [id=c8cd20f7b590464f8f2b92eb85c62795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:17.059092 28006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d802dd1b82e84bba9e99d9139074afbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:17.059115 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d802dd1b82e84bba9e99d9139074afbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:17.062186 28349 tablet_bootstrap.cc:492] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: Bootstrap starting.
I20260925 02:35:17.062539 28350 tablet_bootstrap.cc:492] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7: Bootstrap starting.
I20260925 02:35:17.062618 28348 tablet_bootstrap.cc:492] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Bootstrap starting.
I20260925 02:35:17.063302 28349 tablet_bootstrap.cc:654] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:17.063315 28348 tablet_bootstrap.cc:654] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:17.063315 28350 tablet_bootstrap.cc:654] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:17.063643 28348 log.cc:824] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:17.063643 28350 log.cc:824] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:17.063755 28349 log.cc:824] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:17.064543 28348 tablet_bootstrap.cc:492] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: No bootstrap required, opened a new log
I20260925 02:35:17.064620 28349 tablet_bootstrap.cc:492] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: No bootstrap required, opened a new log
I20260925 02:35:17.064615 28348 ts_tablet_manager.cc:1397] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:35:17.064663 28349 ts_tablet_manager.cc:1397] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:35:17.064677 28350 tablet_bootstrap.cc:492] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7: No bootstrap required, opened a new log
I20260925 02:35:17.064735 28350 ts_tablet_manager.cc:1397] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:35:17.066654 28349 raft_consensus.cc:348] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.066866 28349 raft_consensus.cc:374] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:17.066934 28349 raft_consensus.cc:729] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b585776762446a3ba5cd9fabdb3fd12, State: Initialized, Role: FOLLOWER
I20260925 02:35:17.067068 28349 consensus_queue.cc:260] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.067269 28350 raft_consensus.cc:348] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.067377 28349 ts_tablet_manager.cc:1428] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:17.067401 28334 heartbeater.cc:503] Master 127.10.13.62:34899 was elected leader, sending a full tablet report...
I20260925 02:35:17.067430 28350 raft_consensus.cc:374] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:17.067466 28350 raft_consensus.cc:729] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded31e45ba31495b8ec0b498d54769a7, State: Initialized, Role: FOLLOWER
I20260925 02:35:17.067627 28350 consensus_queue.cc:260] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.067262 28348 raft_consensus.cc:348] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.067831 28348 raft_consensus.cc:374] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:17.067917 28350 ts_tablet_manager.cc:1428] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:17.067983 28072 heartbeater.cc:503] Master 127.10.13.62:34899 was elected leader, sending a full tablet report...
I20260925 02:35:17.067942 28348 raft_consensus.cc:729] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5393eff53fe4110b7b33a18004f7950, State: Initialized, Role: FOLLOWER
I20260925 02:35:17.068444 28348 consensus_queue.cc:260] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.068862 28348 ts_tablet_manager.cc:1428] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:35:17.069003 28203 heartbeater.cc:503] Master 127.10.13.62:34899 was elected leader, sending a full tablet report...
I20260925 02:35:17.109977 28355 raft_consensus.cc:482] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:17.110131 28355 raft_consensus.cc:504] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.110514 28355 leader_election.cc:288] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b585776762446a3ba5cd9fabdb3fd12 (127.10.13.3:46719), c5393eff53fe4110b7b33a18004f7950 (127.10.13.2:39933)
I20260925 02:35:17.113384 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.110725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:46719 (local address 127.10.13.1:37431)
0925 02:35:17.110780 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:17.110793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:17.110888 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:17.111048 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:17.111049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:17.111050 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:17.111124 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.111126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.111756 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.111757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.112503 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.112505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.113119 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.113119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.113198 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.113201 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:17.113266 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:35:17.113308 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:35:17.113384 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.110798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:37431 (local address 127.10.13.3:46719)
0925 02:35:17.110865 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:17.110867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:17.110879 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:35:17.110973 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:17.110973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:17.110984 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:17.111041 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:17.111194 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.111671 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.112595 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.113037 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.113118 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.113129 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:17.113272 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:35:17.113308 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:35:17.113633 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.110912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:36887 (local address 127.10.13.2:39933)
0925 02:35:17.110958 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:17.110960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:17.111100 (+ 140us) server_negotiation.cc:407] Connection header received
0925 02:35:17.111181 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:17.111182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:17.111195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:17.111268 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:17.111419 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.111874 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.112792 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.113218 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.113303 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.113313 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:17.113509 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 02:35:17.113558 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:35:17.113783 28288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d802dd1b82e84bba9e99d9139074afbf" candidate_uuid: "ded31e45ba31495b8ec0b498d54769a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b585776762446a3ba5cd9fabdb3fd12" is_pre_election: true
I20260925 02:35:17.113868 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.110839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:39933 (local address 127.10.13.1:36887)
0925 02:35:17.111028 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:17.111036 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:17.111114 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:17.111270 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:17.111271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:17.111273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:17.111349 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.111351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.111966 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.111967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.112699 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:17.112700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.113319 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.113320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:17.113395 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.113398 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:17.113737 (+ 339us) client_negotiation.cc:240] Negotiation successful
0925 02:35:17.113791 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":115,"threads_started":1}
I20260925 02:35:17.113967 28288 raft_consensus.cc:2454] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded31e45ba31495b8ec0b498d54769a7 in term 0.
I20260925 02:35:17.114260 27959 leader_election.cc:302] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b585776762446a3ba5cd9fabdb3fd12, ded31e45ba31495b8ec0b498d54769a7; no voters:
I20260925 02:35:17.114244 28157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d802dd1b82e84bba9e99d9139074afbf" candidate_uuid: "ded31e45ba31495b8ec0b498d54769a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5393eff53fe4110b7b33a18004f7950" is_pre_election: true
I20260925 02:35:17.114391 28355 raft_consensus.cc:2781] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:35:17.114403 28157 raft_consensus.cc:2454] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded31e45ba31495b8ec0b498d54769a7 in term 0.
I20260925 02:35:17.114446 28355 raft_consensus.cc:482] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:35:17.114530 28355 raft_consensus.cc:3037] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:17.115254 28355 raft_consensus.cc:504] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.115406 28355 leader_election.cc:288] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [CANDIDATE]: Term 1 election: Requested vote from peers 6b585776762446a3ba5cd9fabdb3fd12 (127.10.13.3:46719), c5393eff53fe4110b7b33a18004f7950 (127.10.13.2:39933)
I20260925 02:35:17.115607 28157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d802dd1b82e84bba9e99d9139074afbf" candidate_uuid: "ded31e45ba31495b8ec0b498d54769a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5393eff53fe4110b7b33a18004f7950"
I20260925 02:35:17.115624 28288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d802dd1b82e84bba9e99d9139074afbf" candidate_uuid: "ded31e45ba31495b8ec0b498d54769a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b585776762446a3ba5cd9fabdb3fd12"
I20260925 02:35:17.115685 28157 raft_consensus.cc:3037] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:17.115711 28288 raft_consensus.cc:3037] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:17.116503 28288 raft_consensus.cc:2454] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded31e45ba31495b8ec0b498d54769a7 in term 1.
I20260925 02:35:17.116611 28157 raft_consensus.cc:2454] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded31e45ba31495b8ec0b498d54769a7 in term 1.
I20260925 02:35:17.116712 27959 leader_election.cc:302] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b585776762446a3ba5cd9fabdb3fd12, ded31e45ba31495b8ec0b498d54769a7; no voters:
I20260925 02:35:17.116842 28355 raft_consensus.cc:2781] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:17.117038 28355 raft_consensus.cc:686] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 1 LEADER]: Becoming Leader. State: Replica: ded31e45ba31495b8ec0b498d54769a7, State: Running, Role: LEADER
I20260925 02:35:17.117161 28355 consensus_queue.cc:237] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } }
I20260925 02:35:17.118044 27883 catalog_manager.cc:5495] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ded31e45ba31495b8ec0b498d54769a7 (127.10.13.1). New cstate: current_term: 1 leader_uuid: "ded31e45ba31495b8ec0b498d54769a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded31e45ba31495b8ec0b498d54769a7" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 34693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20260925 02:35:17.135386 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:17.133212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.10.13.1:34693)
0925 02:35:17.133284 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:17.133286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:17.133295 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:17.133334 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:17.133335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:17.133357 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:17.133420 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:17.133536 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.134002 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.134690 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:17.134870 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:17.134917 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:17.135048 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:17.135146 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:17.135206 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:17.135237 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 02:35:17.135253 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 02:35:17.135974 28204 tablet.cc:2369] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:35:17.137378 28157 raft_consensus.cc:1261] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 1 FOLLOWER]: Refusing update from remote peer ded31e45ba31495b8ec0b498d54769a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:17.137461 28288 raft_consensus.cc:1261] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 1 FOLLOWER]: Refusing update from remote peer ded31e45ba31495b8ec0b498d54769a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:17.137751 28355 consensus_queue.cc:1035] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5393eff53fe4110b7b33a18004f7950" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:17.137899 28358 consensus_queue.cc:1035] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b585776762446a3ba5cd9fabdb3fd12" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 46719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:17.141359 28363 mvcc.cc:205] Tried to move back new op lower bound from 7333084025391714304 to 7333084025312612352. Current Snapshot: MvccSnapshot[applied={T|T < 7333084025391714304 or (T in {7333084025391714304})}]
I20260925 02:35:17.142354 27883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:17.142459 27883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:17.143334 27883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3fd1e4d67fa4d4d9f25a6cde5eaca26: Sending DeleteTablet for 3 replicas of tablet d802dd1b82e84bba9e99d9139074afbf
I20260925 02:35:17.143633 28268 tablet_service.cc:1514] Processing DeleteTablet for tablet d802dd1b82e84bba9e99d9139074afbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:17 UTC) from {username='slave'} at 127.0.0.1:51614
I20260925 02:35:17.143725 28006 tablet_service.cc:1514] Processing DeleteTablet for tablet d802dd1b82e84bba9e99d9139074afbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:17 UTC) from {username='slave'} at 127.0.0.1:39408
I20260925 02:35:17.143725 28137 tablet_service.cc:1514] Processing DeleteTablet for tablet d802dd1b82e84bba9e99d9139074afbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:17 UTC) from {username='slave'} at 127.0.0.1:56000
I20260925 02:35:17.143967 28372 tablet_replica.cc:330] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7: stopping tablet replica
I20260925 02:35:17.144039 28370 tablet_replica.cc:330] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: stopping tablet replica
I20260925 02:35:17.144135 28372 raft_consensus.cc:2229] T d802dd1b82e84bba9e99d9139074afbf P ded31e45ba31495b8ec0b498d54769a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:35:17.144165 28370 raft_consensus.cc:2229] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:17.144243 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 27944
I20260925 02:35:17.144285 28370 raft_consensus.cc:2258] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:17.144968 28371 tablet_replica.cc:330] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: stopping tablet replica
I20260925 02:35:17.145103 28371 raft_consensus.cc:2229] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:17.145205 28371 raft_consensus.cc:2258] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:17.145990 28370 ts_tablet_manager.cc:1905] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:17.146567 28371 ts_tablet_manager.cc:1905] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:17.147732 28370 ts_tablet_manager.cc:1918] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:17.147804 28370 log.cc:1196] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/d802dd1b82e84bba9e99d9139074afbf
I20260925 02:35:17.148061 28370 ts_tablet_manager.cc:1939] T d802dd1b82e84bba9e99d9139074afbf P 6b585776762446a3ba5cd9fabdb3fd12: Deleting consensus metadata
I20260925 02:35:17.148262 28371 ts_tablet_manager.cc:1918] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:17.148343 28371 log.cc:1196] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/d802dd1b82e84bba9e99d9139074afbf
I20260925 02:35:17.148567 28371 ts_tablet_manager.cc:1939] T d802dd1b82e84bba9e99d9139074afbf P c5393eff53fe4110b7b33a18004f7950: Deleting consensus metadata
I20260925 02:35:17.148976 27868 catalog_manager.cc:4841] TS c5393eff53fe4110b7b33a18004f7950 (127.10.13.2:39933): tablet d802dd1b82e84bba9e99d9139074afbf (table test-table [id=c8cd20f7b590464f8f2b92eb85c62795]) successfully deleted
W20260925 02:35:17.151080 27868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:34693 (error 108)
I20260925 02:35:17.151491 27868 catalog_manager.cc:4841] TS 6b585776762446a3ba5cd9fabdb3fd12 (127.10.13.3:46719): tablet d802dd1b82e84bba9e99d9139074afbf (table test-table [id=c8cd20f7b590464f8f2b92eb85c62795]) successfully deleted
W20260925 02:35:17.151568 28346 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:17.151419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:34693 (local address 127.0.0.1:39438)
0925 02:35:17.151480 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:17.151529 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:34693: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:35:17.151661 27868 catalog_manager.cc:4584] TS ded31e45ba31495b8ec0b498d54769a7 (127.10.13.1:34693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d802dd1b82e84bba9e99d9139074afbf: Network error: Client connection negotiation failed: client connection to 127.10.13.1:34693: connect: Connection reset by peer (error 104)
I20260925 02:35:17.151757 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 28075
I20260925 02:35:17.157066 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 28206
I20260925 02:35:17.161540 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 27853
2026-09-25T02:35:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (718 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:17 dist-test-slave-q8mg krb5kdc[28376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:17 dist-test-slave-q8mg krb5kdc[28376](info): set up 2 sockets
Sep 25 02:35:17 dist-test-slave-q8mg krb5kdc[28376](info): commencing operation
krb5kdc: starting...
W20260925 02:35:18.191176 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:18.312469 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:42745
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:42745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:18.394187 28389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:18.394454 28389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:18.394495 28389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:18.395685 28389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:18.395762 28389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:18.395788 28389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:18.395828 28389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:18.395856 28389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:18.397375 28389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:42745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:42745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:18.397657 28389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:18.397897 28389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:18.402890 28397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:18.402987 28394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:18.402906 28395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:18.403111 28389 server_base.cc:1034] running on GCE node
I20260925 02:35:18.403347 28389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:18.403589 28389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:18.404733 28389 hybrid_clock.cc:648] HybridClock initialized: now 1790303718404723 us; error 33 us; skew 500 ppm
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:18.407706 28389 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:18.408206 28389 webserver.cc:460] Webserver started at http://127.10.13.62:46595/ using document root <none> and password file <none>
I20260925 02:35:18.408442 28389 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:18.408486 28389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:18.408645 28389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:18.409799 28389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "abff35946dba4e289013df7f558b8294"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "0a58eabd987822d7205db2f7b83ebcad"
server_key_iv: "f42f6f981eddb3bd009bc9205a600813"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.410190 28389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "abff35946dba4e289013df7f558b8294"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "0a58eabd987822d7205db2f7b83ebcad"
server_key_iv: "f42f6f981eddb3bd009bc9205a600813"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.411598 28389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:35:18.412416 28404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.412613 28389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:18.412719 28389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "abff35946dba4e289013df7f558b8294"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "0a58eabd987822d7205db2f7b83ebcad"
server_key_iv: "f42f6f981eddb3bd009bc9205a600813"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.412808 28389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:18.433164 28389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:18.435300 28389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:18.435485 28389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:18.439848 28389 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:42745
I20260925 02:35:18.439896 28456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:42745 every 8 connection(s)
I20260925 02:35:18.440300 28389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:18.441380 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:18.443921 28457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294: Bootstrap starting.
I20260925 02:35:18.444593 28457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:18.444993 28457 log.cc:824] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:18.445767 28457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294: No bootstrap required, opened a new log
I20260925 02:35:18.447572 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 28389
I20260925 02:35:18.447705 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:18.447935 10292 external_mini_cluster.cc:1428] Setting key 2072c097b25208fd0a7798dd92149687
I20260925 02:35:18.447767 28457 raft_consensus.cc:348] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff35946dba4e289013df7f558b8294" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42745 } }
I20260925 02:35:18.448005 28457 raft_consensus.cc:374] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:18.448060 28457 raft_consensus.cc:729] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abff35946dba4e289013df7f558b8294, State: Initialized, Role: FOLLOWER
I20260925 02:35:18.448252 28457 consensus_queue.cc:260] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff35946dba4e289013df7f558b8294" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42745 } }
I20260925 02:35:18.448370 28457 raft_consensus.cc:388] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:18.448673 28457 raft_consensus.cc:482] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:18.448774 28457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:18.449687 28457 raft_consensus.cc:504] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff35946dba4e289013df7f558b8294" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42745 } }
I20260925 02:35:18.449803 28457 leader_election.cc:302] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abff35946dba4e289013df7f558b8294; no voters:
I20260925 02:35:18.449999 28457 leader_election.cc:288] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:18.450078 28462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:18.450278 28457 sys_catalog.cc:564] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:18.450291 28462 raft_consensus.cc:686] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [term 1 LEADER]: Becoming Leader. State: Replica: abff35946dba4e289013df7f558b8294, State: Running, Role: LEADER
I20260925 02:35:18.450497 28462 consensus_queue.cc:237] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff35946dba4e289013df7f558b8294" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42745 } }
I20260925 02:35:18.450884 28463 sys_catalog.cc:455] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abff35946dba4e289013df7f558b8294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff35946dba4e289013df7f558b8294" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42745 } } }
I20260925 02:35:18.450979 28463 sys_catalog.cc:458] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:18.451144 28464 sys_catalog.cc:455] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abff35946dba4e289013df7f558b8294. Latest consensus state: current_term: 1 leader_uuid: "abff35946dba4e289013df7f558b8294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff35946dba4e289013df7f558b8294" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 42745 } } }
I20260925 02:35:18.451202 28464 sys_catalog.cc:458] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:18.452476 28471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:18.453030 28471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:18.454758 28471 catalog_manager.cc:1347] Generated new cluster ID: d634accec7a94b36af3b995ca6e4abf7
I20260925 02:35:18.454813 28471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:18.459789 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:18.448462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.10.13.62:42745)
0925 02:35:18.448653 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:18.448665 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:18.448711 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:35:18.448857 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:18.448865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:18.448955 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:18.449122 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:18.449734 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.451344 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.452153 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.452981 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.453107 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:18.455846 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:18.455851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:18.455852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:18.455867 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:18.458024 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:18.458447 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:18.458451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:18.458451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:18.458567 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:18.458799 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:18.458800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:18.458801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:18.459054 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:18.459219 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:18.459338 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 02:35:18.459483 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260925 02:35:18.484493 28471 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:18.485286 28471 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:18.489248 28471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294: Generated new TSK 0
I20260925 02:35:18.489467 28471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:35:18.522544 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42745
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:18.613950 28483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:18.614162 28483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:18.614187 28483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:18.615338 28483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:18.615379 28483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:18.615430 28483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:35:18.617082 28483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:18.617336 28483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:18.617585 28483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:18.622730 28488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:18.622730 28491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:18.622788 28489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:18.622911 28483 server_base.cc:1034] running on GCE node
I20260925 02:35:18.623215 28483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:18.623462 28483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:18.624605 28483 hybrid_clock.cc:648] HybridClock initialized: now 1790303718624590 us; error 25 us; skew 500 ppm
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:18.627122 28483 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:35:18.627560 28483 webserver.cc:460] Webserver started at http://127.10.13.1:38323/ using document root <none> and password file <none>
I20260925 02:35:18.627786 28483 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:18.627868 28483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:18.628021 28483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:18.629175 28483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "1f1a265b95e949eab16e0af57eb568ce"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "227c90af919e8e7a238c3906d7b0053e"
server_key_iv: "5a662e503ab26b74a7b9aa580185be83"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.629524 28483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "1f1a265b95e949eab16e0af57eb568ce"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "227c90af919e8e7a238c3906d7b0053e"
server_key_iv: "5a662e503ab26b74a7b9aa580185be83"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.630856 28483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:18.631628 28498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.631946 28483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:18.632050 28483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "1f1a265b95e949eab16e0af57eb568ce"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "227c90af919e8e7a238c3906d7b0053e"
server_key_iv: "5a662e503ab26b74a7b9aa580185be83"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.632151 28483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:18.647373 28483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:18.649685 28483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:18.649868 28483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:18.650125 28483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:18.650458 28483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:18.650516 28483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.650557 28483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:18.650588 28483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.656680 28483 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:38061
I20260925 02:35:18.656733 28611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:38061 every 8 connection(s)
I20260925 02:35:18.657070 28483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:35:18.658413 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 28483
I20260925 02:35:18.658538 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:35:18.658758 10292 external_mini_cluster.cc:1428] Setting key 0856ba85bbb4a45009a6132cfd9a2f14
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:18.667546 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:18.657824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:44513 (local address 127.10.13.62:42745)
0925 02:35:18.657898 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:18.657900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:18.658166 (+ 266us) server_negotiation.cc:407] Connection header received
0925 02:35:18.658648 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:18.658649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:18.658666 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:18.658704 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:18.659950 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.660737 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.661809 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.662144 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.662220 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:18.664365 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:18.664368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:18.664368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:18.664377 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:18.666065 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:18.666495 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:18.666496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:18.666497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:18.666562 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:18.666808 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:18.666809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:18.666810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:18.666961 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:18.667047 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:18.667312 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 02:35:18.667407 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:35:18.667558 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:18.657772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:42745 (local address 127.10.13.1:44513)
0925 02:35:18.657936 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:18.657998 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:18.658502 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:18.658893 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:18.658897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:18.659638 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:18.659841 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:18.659847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.660846 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.660847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:18.661721 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:18.661723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.662232 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.662233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:18.662335 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:18.662877 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:18.662886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:18.664267 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:18.666163 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:18.666170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:18.666172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:18.666406 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:18.666637 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:18.666637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:18.666637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:18.666738 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:18.667036 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:18.667041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:18.667160 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 02:35:18.667250 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:35:18.667375 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
I20260925 02:35:18.668444 28612 heartbeater.cc:347] Connected to a master server at 127.10.13.62:42745
I20260925 02:35:18.668560 28612 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:18.668763 28612 heartbeater.cc:511] Master 127.10.13.62:42745 requested a full tablet report, sending...
I20260925 02:35:18.669364 28421 ts_manager.cc:194] Registered new tserver with Master: 1f1a265b95e949eab16e0af57eb568ce (127.10.13.1:38061)
I20260925 02:35:18.670002 28421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:44513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:35:18.689082 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42745
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:18.772065 28617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:18.772276 28617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:18.772300 28617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:18.773586 28617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:18.773630 28617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:18.773658 28617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:35:18.775214 28617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:18.775471 28617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:18.775738 28617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:18.780966 28625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:18.780972 28622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:18.781009 28623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:18.781188 28617 server_base.cc:1034] running on GCE node
I20260925 02:35:18.781405 28617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:18.781613 28617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:18.782811 28617 hybrid_clock.cc:648] HybridClock initialized: now 1790303718782796 us; error 29 us; skew 500 ppm
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:18.785516 28617 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:35:18.786037 28617 webserver.cc:460] Webserver started at http://127.10.13.2:34375/ using document root <none> and password file <none>
I20260925 02:35:18.786296 28617 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:18.786371 28617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:18.786526 28617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:18.787619 28617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "1fada3db180b433c848b24a6062a18e9"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "31a9641887c7cb2f4a7e3e23c789c41f"
server_key_iv: "d972a894d1bb97c901500573946e7558"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.787995 28617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "1fada3db180b433c848b24a6062a18e9"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "31a9641887c7cb2f4a7e3e23c789c41f"
server_key_iv: "d972a894d1bb97c901500573946e7558"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.789454 28617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:18.790236 28632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.790421 28617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:18.790534 28617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "1fada3db180b433c848b24a6062a18e9"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "31a9641887c7cb2f4a7e3e23c789c41f"
server_key_iv: "d972a894d1bb97c901500573946e7558"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.790611 28617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:18.802521 28617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:18.804661 28617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:18.804836 28617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:18.805133 28617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:18.805469 28617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:18.805526 28617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.805560 28617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:18.805605 28617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.811745 28617 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:44977
I20260925 02:35:18.811789 28745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:44977 every 8 connection(s)
I20260925 02:35:18.812129 28617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:35:18.814142 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 28617
I20260925 02:35:18.814231 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:35:18.814428 10292 external_mini_cluster.cc:1428] Setting key 1b834e32adede10560541409eda3ee35
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
I20260925 02:35:18.822567 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:18.812858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:45997 (local address 127.10.13.62:42745)
0925 02:35:18.812946 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:18.812948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:18.813161 (+ 213us) server_negotiation.cc:407] Connection header received
0925 02:35:18.813621 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:18.813622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:18.813633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:18.813686 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:18.814800 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.815419 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.817078 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.817286 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.817378 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:18.819630 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:18.819631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:18.819632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:18.819640 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:18.821116 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:18.821563 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:18.821564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:18.821565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:18.821625 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:18.821892 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:18.821893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:18.821894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:18.822013 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:18.822088 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:18.822391 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 02:35:18.822471 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:35:18.822772 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:18.812854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:42745 (local address 127.10.13.2:45997)
0925 02:35:18.813028 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:18.813068 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:18.813536 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:18.813758 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:18.813761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:18.814402 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:18.814678 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:18.814685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.815513 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.815514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:18.816982 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:18.816985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.817388 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.817389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:18.817531 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:18.818084 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:18.818092 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:18.819531 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:18.821226 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:18.821233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:18.821236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:18.821471 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:18.821691 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:18.821691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:18.821692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:18.821806 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:18.822080 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:18.822088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:18.822252 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 02:35:18.822365 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:35:18.822533 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260925 02:35:18.823293 28746 heartbeater.cc:347] Connected to a master server at 127.10.13.62:42745
I20260925 02:35:18.823405 28746 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:18.823589 28746 heartbeater.cc:511] Master 127.10.13.62:42745 requested a full tablet report, sending...
I20260925 02:35:18.823974 28421 ts_manager.cc:194] Registered new tserver with Master: 1fada3db180b433c848b24a6062a18e9 (127.10.13.2:44977)
I20260925 02:35:18.824404 28421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:45997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:35:18.839764 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42745
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:18.922184 28751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:18.922394 28751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:18.922416 28751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:18.923588 28751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:18.923631 28751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:18.923658 28751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:35:18.925191 28751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:18.925457 28751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:18.925707 28751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:18.931119 28757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:18.931130 28756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:18.931315 28751 server_base.cc:1034] running on GCE node
W20260925 02:35:18.931130 28759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:18.931576 28751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:18.931847 28751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:18.933012 28751 hybrid_clock.cc:648] HybridClock initialized: now 1790303718932989 us; error 35 us; skew 500 ppm
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:18.935679 28751 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:35:18.936139 28751 webserver.cc:460] Webserver started at http://127.10.13.3:36343/ using document root <none> and password file <none>
I20260925 02:35:18.936393 28751 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:18.936462 28751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:18.936609 28751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:18.937724 28751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "409d1c4255294065a713ffa6e483037d"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "b534c44cae473644d1b373cf12540538"
server_key_iv: "702d96404dbe2965c582aed2febd357a"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.938092 28751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "409d1c4255294065a713ffa6e483037d"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "b534c44cae473644d1b373cf12540538"
server_key_iv: "702d96404dbe2965c582aed2febd357a"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.939447 28751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:35:18.940208 28766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.940379 28751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:35:18.940490 28751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "409d1c4255294065a713ffa6e483037d"
format_stamp: "Formatted at 2026-09-25 02:35:18 on dist-test-slave-q8mg"
server_key: "b534c44cae473644d1b373cf12540538"
server_key_iv: "702d96404dbe2965c582aed2febd357a"
server_key_version: "encryptionkey@0"
I20260925 02:35:18.940567 28751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:18.966226 28751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:18.968715 28751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:18.968945 28751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:18.969205 28751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:18.969537 28751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:18.969591 28751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.969636 28751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:18.969669 28751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:18.975620 28751 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:38299
I20260925 02:35:18.975699 28882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:38299 every 8 connection(s)
I20260925 02:35:18.976039 28751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:35:18 dist-test-slave-q8mg krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303718, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:18.984982 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:18.976825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:45329 (local address 127.10.13.62:42745)
0925 02:35:18.976931 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:18.976934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:18.977182 (+ 248us) server_negotiation.cc:407] Connection header received
0925 02:35:18.977742 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:18.977743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:18.977753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:18.977808 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:18.978777 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.979283 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.980248 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.980433 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.980483 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:18.982593 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:18.982595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:18.982595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:18.982602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:18.983775 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:18.984136 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:18.984137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:18.984137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:18.984187 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:18.984399 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:18.984399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:18.984400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:18.984502 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:18.984552 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:18.984790 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 02:35:18.984860 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:35:18.984997 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 28751
I20260925 02:35:18.985085 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:35:18.985239 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:18.976791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:42745 (local address 127.10.13.3:45329)
0925 02:35:18.977023 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:18.977061 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:18.977645 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:18.977867 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:18.977870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:18.978470 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:18.978673 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:18.978678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.979368 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.979369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:18.980174 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:18.980177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.980506 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:18.980507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:18.980614 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:18.981150 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:18.981159 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:18.982506 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:18.983846 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:18.983853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:18.983855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:18.984071 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:18.984247 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:18.984247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:18.984248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:18.984347 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:18.984556 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:18.984560 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:18.984675 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 02:35:18.984757 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:35:18.984973 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":150,"threads_started":1}
I20260925 02:35:18.985265 10292 external_mini_cluster.cc:1428] Setting key 9f1eee66846d1c6efb9959e5387e2f12
I20260925 02:35:18.985817 28883 heartbeater.cc:347] Connected to a master server at 127.10.13.62:42745
I20260925 02:35:18.985948 28883 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:18.986156 28883 heartbeater.cc:511] Master 127.10.13.62:42745 requested a full tablet report, sending...
I20260925 02:35:18.986509 28420 ts_manager.cc:194] Registered new tserver with Master: 409d1c4255294065a713ffa6e483037d (127.10.13.3:38299)
I20260925 02:35:18.986850 28420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:45329
I20260925 02:35:18.997161 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:35:19.004843 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:18.998471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.10.13.62:42745)
0925 02:35:18.998537 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:18.998539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:18.998679 (+ 140us) server_negotiation.cc:407] Connection header received
0925 02:35:18.998717 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:18.998717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:18.998726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:18.998783 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:18.999313 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:18.999778 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.000497 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.000681 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.000737 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.002150 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:19.002151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:19.002151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:19.002157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:19.003367 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:19.003768 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:19.003769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:19.003770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:19.003835 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:19.004064 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:19.004065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:19.004066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:19.004211 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:19.004324 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.004532 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.004663 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:35:19.007244 28420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:35:19.009557 28420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:35:19.018394 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.014192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.10.13.3:38299)
0925 02:35:19.014619 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.014624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.014635 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:35:19.014680 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.014682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.014715 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.015051 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.015155 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.016248 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.017387 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.017996 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.018084 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.018146 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.018275 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.018310 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":101,"threads_started":1}
I20260925 02:35:19.018384 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.014071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:38299 (local address 127.0.0.1:52992)
0925 02:35:19.014200 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.014247 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.014320 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.014844 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.014847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.014850 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.014971 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.014974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.016371 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.016372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.017275 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.017277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.018086 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.018087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.018165 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.018178 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.018250 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.018303 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260925 02:35:19.018641 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.014280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:38061 (local address 127.0.0.1:47884)
0925 02:35:19.014406 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.014415 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.014454 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.014859 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.014860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.014862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.014977 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.014979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.016088 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.016088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.017482 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.017485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.018408 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.018409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.018473 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.018477 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.018507 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.018555 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":50,"thread_start_us":88,"threads_started":1}
I20260925 02:35:19.018846 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.014607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.10.13.2:44977)
0925 02:35:19.014802 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.014806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.014844 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:35:19.014925 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.014927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.014943 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.015022 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.015229 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.016323 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.017625 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.018295 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.018370 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.018426 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.018679 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.018727 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260925 02:35:19.018643 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.014485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:44977 (local address 127.0.0.1:56224)
0925 02:35:19.014792 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.014802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.014857 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.015107 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.015108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.015110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.015149 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.015150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.016419 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.016419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.017544 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.017546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.018455 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.018456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.018508 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.018511 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.018554 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.018583 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":18,"thread_start_us":70,"threads_started":1}
I20260925 02:35:19.019061 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.014410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.10.13.1:38061)
0925 02:35:19.014660 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.014666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.014676 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:35:19.014718 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.014720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.014735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.014954 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.015040 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.015989 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.017584 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.018295 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.018818 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.018888 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.018915 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.018963 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20260925 02:35:19.019470 28815 tablet_service.cc:1467] Processing CreateTablet for tablet 6be1dbd1b5194c4098dde93d8f566eb4 (DEFAULT_TABLE table=test-table [id=aa5b9eac24fe4a0095980488d8432575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:19.019660 28546 tablet_service.cc:1467] Processing CreateTablet for tablet 6be1dbd1b5194c4098dde93d8f566eb4 (DEFAULT_TABLE table=test-table [id=aa5b9eac24fe4a0095980488d8432575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:19.019658 28680 tablet_service.cc:1467] Processing CreateTablet for tablet 6be1dbd1b5194c4098dde93d8f566eb4 (DEFAULT_TABLE table=test-table [id=aa5b9eac24fe4a0095980488d8432575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:19.019814 28815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be1dbd1b5194c4098dde93d8f566eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:19.019985 28680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be1dbd1b5194c4098dde93d8f566eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:19.019985 28546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be1dbd1b5194c4098dde93d8f566eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:19.022852 28900 tablet_bootstrap.cc:492] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: Bootstrap starting.
I20260925 02:35:19.023238 28902 tablet_bootstrap.cc:492] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: Bootstrap starting.
I20260925 02:35:19.023624 28900 tablet_bootstrap.cc:654] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:19.023990 28902 tablet_bootstrap.cc:654] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:19.024286 28900 log.cc:824] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:19.024608 28902 log.cc:824] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:19.024962 28901 tablet_bootstrap.cc:492] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce: Bootstrap starting.
I20260925 02:35:19.025188 28900 tablet_bootstrap.cc:492] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: No bootstrap required, opened a new log
I20260925 02:35:19.025297 28900 ts_tablet_manager.cc:1397] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:35:19.025398 28902 tablet_bootstrap.cc:492] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: No bootstrap required, opened a new log
I20260925 02:35:19.025465 28902 ts_tablet_manager.cc:1397] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:35:19.025636 28901 tablet_bootstrap.cc:654] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:19.026225 28901 log.cc:824] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:19.026921 28901 tablet_bootstrap.cc:492] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce: No bootstrap required, opened a new log
I20260925 02:35:19.027012 28901 ts_tablet_manager.cc:1397] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:35:19.027309 28900 raft_consensus.cc:348] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
I20260925 02:35:19.027403 28902 raft_consensus.cc:348] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
I20260925 02:35:19.027509 28900 raft_consensus.cc:374] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:19.027550 28902 raft_consensus.cc:374] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:19.027550 28900 raft_consensus.cc:729] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409d1c4255294065a713ffa6e483037d, State: Initialized, Role: FOLLOWER
I20260925 02:35:19.027626 28902 raft_consensus.cc:729] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fada3db180b433c848b24a6062a18e9, State: Initialized, Role: FOLLOWER
I20260925 02:35:19.027678 28900 consensus_queue.cc:260] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
I20260925 02:35:19.027758 28902 consensus_queue.cc:260] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
I20260925 02:35:19.027982 28900 ts_tablet_manager.cc:1428] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 02:35:19.028048 28902 ts_tablet_manager.cc:1428] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:35:19.028093 28746 heartbeater.cc:503] Master 127.10.13.62:42745 was elected leader, sending a full tablet report...
I20260925 02:35:19.028180 28883 heartbeater.cc:503] Master 127.10.13.62:42745 was elected leader, sending a full tablet report...
I20260925 02:35:19.028972 28901 raft_consensus.cc:348] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
I20260925 02:35:19.029152 28901 raft_consensus.cc:374] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:19.029225 28901 raft_consensus.cc:729] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1a265b95e949eab16e0af57eb568ce, State: Initialized, Role: FOLLOWER
I20260925 02:35:19.029358 28901 consensus_queue.cc:260] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
I20260925 02:35:19.029680 28901 ts_tablet_manager.cc:1428] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:19.029742 28612 heartbeater.cc:503] Master 127.10.13.62:42745 was elected leader, sending a full tablet report...
W20260925 02:35:19.062721 28747 tablet.cc:2369] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:35:19.157321 28908 raft_consensus.cc:482] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:19.157452 28908 raft_consensus.cc:504] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
W20260925 02:35:19.157781 28613 tablet.cc:2369] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:35:19.157887 28908 leader_election.cc:288] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 409d1c4255294065a713ffa6e483037d (127.10.13.3:38299), 1fada3db180b433c848b24a6062a18e9 (127.10.13.2:44977)
I20260925 02:35:19.161345 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.158351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:57637 (local address 127.10.13.2:44977)
0925 02:35:19.158400 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.158402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.158457 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:35:19.158518 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.158519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.158529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.158572 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.158739 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.159444 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.160593 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.160994 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.161049 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.161058 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.161226 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.161270 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:35:19.161365 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.158236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:44977 (local address 127.10.13.1:57637)
0925 02:35:19.158388 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.158396 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.158461 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.158605 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.158606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.158608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.158671 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.158672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.159541 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.159542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.160482 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.160485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.161074 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.161076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.161151 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.161156 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.161204 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.161247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260925 02:35:19.161375 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.158215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:35583 (local address 127.10.13.3:38299)
0925 02:35:19.158299 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.158302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.158312 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:35:19.158345 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.158346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.158359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.158437 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.158635 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.159421 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.160547 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.161040 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.161112 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.161124 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.161264 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.161307 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:35:19.161376 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.158100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:38299 (local address 127.10.13.1:35583)
0925 02:35:19.158164 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.158178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.158228 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.158445 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.158446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.158448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.158526 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.158528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.159510 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.159511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.160453 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.160456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.161114 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.161115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.161181 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.161184 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.161254 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.161294 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":32}
I20260925 02:35:19.161911 28700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be1dbd1b5194c4098dde93d8f566eb4" candidate_uuid: "1f1a265b95e949eab16e0af57eb568ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fada3db180b433c848b24a6062a18e9" is_pre_election: true
I20260925 02:35:19.161926 28837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be1dbd1b5194c4098dde93d8f566eb4" candidate_uuid: "1f1a265b95e949eab16e0af57eb568ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409d1c4255294065a713ffa6e483037d" is_pre_election: true
I20260925 02:35:19.162075 28700 raft_consensus.cc:2454] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1a265b95e949eab16e0af57eb568ce in term 0.
I20260925 02:35:19.162075 28837 raft_consensus.cc:2454] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1a265b95e949eab16e0af57eb568ce in term 0.
I20260925 02:35:19.162300 28500 leader_election.cc:302] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1a265b95e949eab16e0af57eb568ce, 409d1c4255294065a713ffa6e483037d; no voters:
I20260925 02:35:19.162441 28908 raft_consensus.cc:2781] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:35:19.162499 28908 raft_consensus.cc:482] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:35:19.162525 28908 raft_consensus.cc:3037] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:19.163201 28908 raft_consensus.cc:504] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
I20260925 02:35:19.163328 28908 leader_election.cc:288] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [CANDIDATE]: Term 1 election: Requested vote from peers 409d1c4255294065a713ffa6e483037d (127.10.13.3:38299), 1fada3db180b433c848b24a6062a18e9 (127.10.13.2:44977)
I20260925 02:35:19.163491 28837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be1dbd1b5194c4098dde93d8f566eb4" candidate_uuid: "1f1a265b95e949eab16e0af57eb568ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "409d1c4255294065a713ffa6e483037d"
I20260925 02:35:19.163551 28700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be1dbd1b5194c4098dde93d8f566eb4" candidate_uuid: "1f1a265b95e949eab16e0af57eb568ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fada3db180b433c848b24a6062a18e9"
I20260925 02:35:19.163566 28837 raft_consensus.cc:3037] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:19.163645 28700 raft_consensus.cc:3037] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:19.164314 28837 raft_consensus.cc:2454] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1a265b95e949eab16e0af57eb568ce in term 1.
I20260925 02:35:19.164469 28700 raft_consensus.cc:2454] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1a265b95e949eab16e0af57eb568ce in term 1.
I20260925 02:35:19.164481 28500 leader_election.cc:302] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1a265b95e949eab16e0af57eb568ce, 409d1c4255294065a713ffa6e483037d; no voters:
I20260925 02:35:19.164584 28908 raft_consensus.cc:2781] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:19.164740 28908 raft_consensus.cc:686] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 1 LEADER]: Becoming Leader. State: Replica: 1f1a265b95e949eab16e0af57eb568ce, State: Running, Role: LEADER
I20260925 02:35:19.164816 28908 consensus_queue.cc:237] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } }
I20260925 02:35:19.165700 28420 catalog_manager.cc:5495] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f1a265b95e949eab16e0af57eb568ce (127.10.13.1). New cstate: current_term: 1 leader_uuid: "1f1a265b95e949eab16e0af57eb568ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f1a265b95e949eab16e0af57eb568ce" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 38061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 } health_report { overall_health: UNKNOWN } } }
I20260925 02:35:19.200595 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.198085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.10.13.1:38061)
0925 02:35:19.198167 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.198170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.198179 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:19.198225 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.198226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.198249 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.198330 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:19.198491 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.199008 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.199796 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.199991 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.200084 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.200240 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:19.200332 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:19.200446 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.200476 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.200485 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:35:19.202582 28837 raft_consensus.cc:1261] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 1 FOLLOWER]: Refusing update from remote peer 1f1a265b95e949eab16e0af57eb568ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:19.202608 28700 raft_consensus.cc:1261] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 1 FOLLOWER]: Refusing update from remote peer 1f1a265b95e949eab16e0af57eb568ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:19.203006 28910 consensus_queue.cc:1035] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "409d1c4255294065a713ffa6e483037d" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:19.203142 28908 consensus_queue.cc:1035] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fada3db180b433c848b24a6062a18e9" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:19.205462 28913 mvcc.cc:205] Tried to move back new op lower bound from 7333084033850908672 to 7333084033700003840. Current Snapshot: MvccSnapshot[applied={T|T < 7333084033850908672}]
I20260925 02:35:19.206772 28915 mvcc.cc:205] Tried to move back new op lower bound from 7333084033850908672 to 7333084033700003840. Current Snapshot: MvccSnapshot[applied={T|T < 7333084033850908672 or (T in {7333084033850908672})}]
I20260925 02:35:19.207891 28420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:19.208007 28420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:19.208734 28420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abff35946dba4e289013df7f558b8294: Sending DeleteTablet for 3 replicas of tablet 6be1dbd1b5194c4098dde93d8f566eb4
I20260925 02:35:19.209084 28546 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be1dbd1b5194c4098dde93d8f566eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:19 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:47884
I20260925 02:35:19.209134 28815 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be1dbd1b5194c4098dde93d8f566eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:19 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:52992
I20260925 02:35:19.209298 28923 tablet_replica.cc:330] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce: stopping tablet replica
I20260925 02:35:19.209388 28923 raft_consensus.cc:2229] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:35:19.209391 28924 tablet_replica.cc:330] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: stopping tablet replica
I20260925 02:35:19.209482 28924 raft_consensus.cc:2229] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:19.209530 28923 raft_consensus.cc:2258] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1f1a265b95e949eab16e0af57eb568ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:19.209613 28680 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be1dbd1b5194c4098dde93d8f566eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:19 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:56224
I20260925 02:35:19.209721 28924 raft_consensus.cc:2258] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:19.209909 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 28483
I20260925 02:35:19.210157 28924 ts_tablet_manager.cc:1905] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:19.211889 28924 ts_tablet_manager.cc:1918] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:19.211952 28924 log.cc:1196] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/6be1dbd1b5194c4098dde93d8f566eb4
I20260925 02:35:19.212180 28924 ts_tablet_manager.cc:1939] T 6be1dbd1b5194c4098dde93d8f566eb4 P 409d1c4255294065a713ffa6e483037d: Deleting consensus metadata
I20260925 02:35:19.212527 28406 catalog_manager.cc:4841] TS 409d1c4255294065a713ffa6e483037d (127.10.13.3:38299): tablet 6be1dbd1b5194c4098dde93d8f566eb4 (table test-table [id=aa5b9eac24fe4a0095980488d8432575]) successfully deleted
I20260925 02:35:19.215167 28925 tablet_replica.cc:330] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: stopping tablet replica
I20260925 02:35:19.215296 28925 raft_consensus.cc:2229] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:19.215411 28925 raft_consensus.cc:2258] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:19.215864 28925 ts_tablet_manager.cc:1905] T 6be1dbd1b5194c4098dde93d8f566eb4 P 1fada3db180b433c848b24a6062a18e9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:35:19.217097 28406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:38061 (error 108)
I20260925 02:35:19.217308 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 28617
I20260925 02:35:19.218218 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.217488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:38061 (local address 127.0.0.1:47902)
0925 02:35:19.218147 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.218189 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:38061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634}
W20260925 02:35:19.218351 28406 catalog_manager.cc:4584] TS 1f1a265b95e949eab16e0af57eb568ce (127.10.13.1:38061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6be1dbd1b5194c4098dde93d8f566eb4: Network error: Client connection negotiation failed: client connection to 127.10.13.1:38061: connect: Connection refused (error 111)
I20260925 02:35:19.223663 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 28751
I20260925 02:35:19.223686 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.223548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:44977 (local address 127.0.0.1:56234)
0925 02:35:19.223615 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.223649 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.2:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:35:19.228525 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 28389
2026-09-25T02:35:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2068 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T02:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:19Z Disabled control of system clock
I20260925 02:35:19.256688 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:32787
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:32787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:19.338477 28930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:19.338711 28930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:19.338730 28930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:19.339823 28930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:19.339860 28930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:19.339874 28930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:19.339886 28930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:19.339900 28930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:19.341343 28930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:32787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:32787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:19.341557 28930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:19.341804 28930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:19.346695 28935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:19.346681 28938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:19.346932 28930 server_base.cc:1034] running on GCE node
W20260925 02:35:19.346681 28936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:19.347244 28930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:19.347544 28930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:19.348729 28930 hybrid_clock.cc:648] HybridClock initialized: now 1790303719348722 us; error 38 us; skew 500 ppm
I20260925 02:35:19.350052 28930 webserver.cc:460] Webserver started at http://127.10.13.62:41977/ using document root <none> and password file <none>
I20260925 02:35:19.350306 28930 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:19.350379 28930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:19.350535 28930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:19.351642 28930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "00a36dadaa1d451b9ad4cbdef56750df"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "6d7660ea76aab139198f05d88b8745b2"
server_key_iv: "b4b162981c5d780521178a32ad0ab7f6"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.352020 28930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "00a36dadaa1d451b9ad4cbdef56750df"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "6d7660ea76aab139198f05d88b8745b2"
server_key_iv: "b4b162981c5d780521178a32ad0ab7f6"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.353500 28930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:19.354338 28944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.354538 28930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:35:19.354655 28930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "00a36dadaa1d451b9ad4cbdef56750df"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "6d7660ea76aab139198f05d88b8745b2"
server_key_iv: "b4b162981c5d780521178a32ad0ab7f6"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.354736 28930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:19.363574 28930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:19.363905 28930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:19.364070 28930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:19.368181 28930 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:32787
I20260925 02:35:19.368216 28996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:32787 every 8 connection(s)
I20260925 02:35:19.368608 28930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:19.369678 28997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:19.371057 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 28930
I20260925 02:35:19.371184 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:19.371382 10292 external_mini_cluster.cc:1428] Setting key 475c4ac05c809b1333a52ff2a1ad6f98
I20260925 02:35:19.372442 28997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df: Bootstrap starting.
I20260925 02:35:19.373198 28997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:19.373512 28997 log.cc:824] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:19.374255 28997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df: No bootstrap required, opened a new log
I20260925 02:35:19.376619 28997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a36dadaa1d451b9ad4cbdef56750df" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 32787 } }
I20260925 02:35:19.376782 28997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:19.376833 28997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a36dadaa1d451b9ad4cbdef56750df, State: Initialized, Role: FOLLOWER
I20260925 02:35:19.377012 28997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a36dadaa1d451b9ad4cbdef56750df" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 32787 } }
I20260925 02:35:19.377116 28997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:19.377185 28997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:19.377246 28997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:19.377388 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.371968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.10.13.62:32787)
0925 02:35:19.372166 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.372174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.372199 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:35:19.372639 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.372647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.372713 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.372831 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:19.373141 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.374116 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.375230 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.375654 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.375746 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.376507 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:19.376511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:19.376511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:19.376675 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:19.376709 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:19.376719 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:19.376770 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.376995 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.377081 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 02:35:19.377961 28997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a36dadaa1d451b9ad4cbdef56750df" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 32787 } }
I20260925 02:35:19.378074 28997 leader_election.cc:302] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a36dadaa1d451b9ad4cbdef56750df; no voters:
I20260925 02:35:19.378242 28997 leader_election.cc:288] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:19.378268 29002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:19.378441 29002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [term 1 LEADER]: Becoming Leader. State: Replica: 00a36dadaa1d451b9ad4cbdef56750df, State: Running, Role: LEADER
I20260925 02:35:19.378549 28997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:19.378611 29002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a36dadaa1d451b9ad4cbdef56750df" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 32787 } }
I20260925 02:35:19.379042 29004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a36dadaa1d451b9ad4cbdef56750df. Latest consensus state: current_term: 1 leader_uuid: "00a36dadaa1d451b9ad4cbdef56750df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a36dadaa1d451b9ad4cbdef56750df" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 32787 } } }
I20260925 02:35:19.379163 29004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:19.379347 29011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:19.379037 29003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a36dadaa1d451b9ad4cbdef56750df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a36dadaa1d451b9ad4cbdef56750df" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 32787 } } }
I20260925 02:35:19.379693 29003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:19.379917 29011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:19.381503 29011 catalog_manager.cc:1347] Generated new cluster ID: 2fc45d1d65244b70b94b1c521b5e3fd4
I20260925 02:35:19.381559 29011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:19.392448 29011 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:19.393224 29011 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:19.399925 29011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df: Generated new TSK 0
I20260925 02:35:19.400120 29011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:19.405941 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:32787
--builtin_ntp_servers=127.10.13.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:19.491530 29021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:19.491770 29021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:19.491804 29021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:19.492995 29021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:19.493050 29021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:19.493091 29021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:35:19.494429 29021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:19.494699 29021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:19.495002 29021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:19.501036 29027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:19.501051 29029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:19.501051 29026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:19.501430 29021 server_base.cc:1034] running on GCE node
I20260925 02:35:19.501648 29021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:19.501891 29021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:19.503067 29021 hybrid_clock.cc:648] HybridClock initialized: now 1790303719503055 us; error 34 us; skew 500 ppm
I20260925 02:35:19.504372 29021 webserver.cc:460] Webserver started at http://127.10.13.1:46655/ using document root <none> and password file <none>
I20260925 02:35:19.504637 29021 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:19.504707 29021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:19.504861 29021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:19.505991 29021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "0fdb1253638343be82436b92ab480aa6"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "507926718f15a338eda7909d4689e8f9"
server_key_iv: "2286a6dbeeca0dcc667290ae3cb75607"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.506354 29021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "0fdb1253638343be82436b92ab480aa6"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "507926718f15a338eda7909d4689e8f9"
server_key_iv: "2286a6dbeeca0dcc667290ae3cb75607"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.507712 29021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:19.508527 29035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.508746 29021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:35:19.508841 29021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "0fdb1253638343be82436b92ab480aa6"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "507926718f15a338eda7909d4689e8f9"
server_key_iv: "2286a6dbeeca0dcc667290ae3cb75607"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.509004 29021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:19.521622 29021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:19.521974 29021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:19.522137 29021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:19.522370 29021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:19.522708 29021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:19.522765 29021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.522809 29021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:19.522836 29021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.528928 29021 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:33239
I20260925 02:35:19.528954 29148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:33239 every 8 connection(s)
I20260925 02:35:19.529313 29021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:35:19.531419 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 29021
I20260925 02:35:19.531513 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:35:19.531780 10292 external_mini_cluster.cc:1428] Setting key 7a530c5ba53f8912c78dbab76ca3c2d3
I20260925 02:35:19.532802 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:32787
--builtin_ntp_servers=127.10.13.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:35:19.535203 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.530053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:33565 (local address 127.10.13.62:32787)
0925 02:35:19.530100 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.530102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.530326 (+ 224us) server_negotiation.cc:407] Connection header received
0925 02:35:19.530850 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.530852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.530865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.530922 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:19.531744 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.532461 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.533573 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.533794 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.533881 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.534842 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:19.534844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:19.534845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:19.534851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:19.534874 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:19.534880 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:19.534941 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.535055 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.535085 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:35:19.535409 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.530036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:32787 (local address 127.10.13.1:33565)
0925 02:35:19.530217 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.530252 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.530727 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.530986 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.530990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.530998 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:19.531642 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.531649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.532541 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.532543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.533469 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.533472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.533914 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.533915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.534101 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.534692 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:19.534692 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:19.534722 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:19.534746 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:19.534970 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:19.534977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:19.534980 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.535054 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.535141 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260925 02:35:19.535957 29149 heartbeater.cc:347] Connected to a master server at 127.10.13.62:32787
I20260925 02:35:19.536070 29149 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:19.536264 29149 heartbeater.cc:511] Master 127.10.13.62:32787 requested a full tablet report, sending...
I20260925 02:35:19.536828 28961 ts_manager.cc:194] Registered new tserver with Master: 0fdb1253638343be82436b92ab480aa6 (127.10.13.1:33239)
I20260925 02:35:19.537529 28961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.1:33565
W20260925 02:35:19.615955 29152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:19.616165 29152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:19.616186 29152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:19.617450 29152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:19.617491 29152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:19.617517 29152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:35:19.618876 29152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:19.619133 29152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:19.619449 29152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:19.624377 29158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:19.624358 29160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:19.624387 29157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:19.624571 29152 server_base.cc:1034] running on GCE node
I20260925 02:35:19.624925 29152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:19.625191 29152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:19.626341 29152 hybrid_clock.cc:648] HybridClock initialized: now 1790303719626313 us; error 43 us; skew 500 ppm
I20260925 02:35:19.627652 29152 webserver.cc:460] Webserver started at http://127.10.13.2:45709/ using document root <none> and password file <none>
I20260925 02:35:19.627985 29152 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:19.628034 29152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:19.628211 29152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:19.629390 29152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "2665aa6704894b51a31d7ac422462b95"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "7823b7fc1903ec1e46367ae179d41099"
server_key_iv: "a9ec67969075324fa0fd8b5d3a9800eb"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.629781 29152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "2665aa6704894b51a31d7ac422462b95"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "7823b7fc1903ec1e46367ae179d41099"
server_key_iv: "a9ec67969075324fa0fd8b5d3a9800eb"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.631063 29152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:19.631798 29166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.631992 29152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:19.632087 29152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "2665aa6704894b51a31d7ac422462b95"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "7823b7fc1903ec1e46367ae179d41099"
server_key_iv: "a9ec67969075324fa0fd8b5d3a9800eb"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.632161 29152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:19.647753 29152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:19.648089 29152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:19.648241 29152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:19.648488 29152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:19.648859 29152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:19.648985 29152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.649010 29152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:19.649053 29152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.654719 29152 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:32973
I20260925 02:35:19.654789 29279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:32973 every 8 connection(s)
I20260925 02:35:19.655157 29152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:35:19.658419 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 29152
I20260925 02:35:19.658537 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:35:19.658756 10292 external_mini_cluster.cc:1428] Setting key 52099dd63329c6346c1c50cb53fe3ab3
I20260925 02:35:19.659792 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:32787
--builtin_ntp_servers=127.10.13.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:35:19.661273 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.655938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:53541 (local address 127.10.13.62:32787)
0925 02:35:19.656008 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.656010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.656262 (+ 252us) server_negotiation.cc:407] Connection header received
0925 02:35:19.656748 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.656749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.656759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.656815 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:19.657602 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.658161 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.659627 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.659810 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.659868 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.660942 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:19.660946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:19.660946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:19.660956 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:19.660970 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:19.660974 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:19.661019 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.661141 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.661164 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:35:19.661520 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.655934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:32787 (local address 127.10.13.2:53541)
0925 02:35:19.656122 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.656166 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.656665 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.656936 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.656941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.656951 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:19.657523 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.657528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.658279 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.658280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.659540 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.659544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.660045 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.660046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.660210 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.660758 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:19.660758 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:19.660798 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:19.660829 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:19.661051 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:19.661058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:19.661061 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.661134 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.661253 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260925 02:35:19.662040 29280 heartbeater.cc:347] Connected to a master server at 127.10.13.62:32787
I20260925 02:35:19.662153 29280 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:19.662362 29280 heartbeater.cc:511] Master 127.10.13.62:32787 requested a full tablet report, sending...
I20260925 02:35:19.662753 28961 ts_manager.cc:194] Registered new tserver with Master: 2665aa6704894b51a31d7ac422462b95 (127.10.13.2:32973)
I20260925 02:35:19.663159 28961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.2:53541
W20260925 02:35:19.742309 29283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:19.742515 29283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:19.742537 29283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:19.743670 29283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:19.743709 29283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:19.743736 29283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:35:19.745230 29283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:19.745484 29283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:19.745780 29283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:35:19.750855 29283 server_base.cc:1034] running on GCE node
W20260925 02:35:19.750784 29288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:19.750785 29291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:19.750794 29289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:19.751243 29283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:19.751458 29283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:19.752595 29283 hybrid_clock.cc:648] HybridClock initialized: now 1790303719752575 us; error 36 us; skew 500 ppm
I20260925 02:35:19.753847 29283 webserver.cc:460] Webserver started at http://127.10.13.3:43057/ using document root <none> and password file <none>
I20260925 02:35:19.754063 29283 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:19.754107 29283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:19.754195 29283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:19.755254 29283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "5d9a9807aebf4cd7bd6279324a027bac"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "92bba5ca2277bf30200ee34ce6301f22"
server_key_iv: "59da034f37e510d412b8d3a96f5e4bc4"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.755566 29283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "5d9a9807aebf4cd7bd6279324a027bac"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "92bba5ca2277bf30200ee34ce6301f22"
server_key_iv: "59da034f37e510d412b8d3a96f5e4bc4"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.756806 29283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:19.757637 29297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.757810 29283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:19.757921 29283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "5d9a9807aebf4cd7bd6279324a027bac"
format_stamp: "Formatted at 2026-09-25 02:35:19 on dist-test-slave-q8mg"
server_key: "92bba5ca2277bf30200ee34ce6301f22"
server_key_iv: "59da034f37e510d412b8d3a96f5e4bc4"
server_key_version: "encryptionkey@0"
I20260925 02:35:19.757999 29283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:19.765488 29283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:19.765769 29283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:19.765918 29283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:19.766135 29283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:19.766463 29283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:19.766520 29283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.766553 29283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:19.766579 29283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:19.772644 29283 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:40813
I20260925 02:35:19.772691 29410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:40813 every 8 connection(s)
I20260925 02:35:19.773041 29283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:35:19.775532 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 29283
I20260925 02:35:19.775666 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:35:19.775869 10292 external_mini_cluster.cc:1428] Setting key b8918fe0085d951a0a24c966cc1a3508
I20260925 02:35:19.778688 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.773759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:54103 (local address 127.10.13.62:32787)
0925 02:35:19.773815 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.773818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.774099 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:35:19.774588 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.774589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.774606 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.774664 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:19.775391 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.776167 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.777397 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.777580 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.777635 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.778407 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:19.778408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:19.778409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:19.778413 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:19.778426 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:19.778430 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:19.778473 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.778606 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.778628 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:35:19.778911 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.773771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:32787 (local address 127.10.13.3:54103)
0925 02:35:19.773980 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.774015 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.774494 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.774742 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.774745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.774754 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:19.775257 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.775261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.776272 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.776273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.777328 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.777331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.777656 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.777656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.777765 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.778279 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:19.778279 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:19.778307 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:19.778330 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:19.778501 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:19.778506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:19.778508 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.778602 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.778682 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260925 02:35:19.779449 29411 heartbeater.cc:347] Connected to a master server at 127.10.13.62:32787
I20260925 02:35:19.779562 29411 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:19.779790 29411 heartbeater.cc:511] Master 127.10.13.62:32787 requested a full tablet report, sending...
I20260925 02:35:19.780184 28961 ts_manager.cc:194] Registered new tserver with Master: 5d9a9807aebf4cd7bd6279324a027bac (127.10.13.3:40813)
I20260925 02:35:19.780505 28961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.3:54103
I20260925 02:35:19.787855 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:35:19.791998 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.789287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.10.13.62:32787)
0925 02:35:19.789334 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.789336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.789417 (+ 81us) server_negotiation.cc:407] Connection header received
0925 02:35:19.789460 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.789460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.789469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.789518 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:19.789677 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.790156 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.790847 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.791019 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.791068 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.791697 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:19.791698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:19.791698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:19.791703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:19.791715 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:19.791719 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:19.791796 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.791863 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.791873 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:35:19.793612 28961 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:35:19.795115 28961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:35:19.801566 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.797666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.10.13.3:40813)
0925 02:35:19.797868 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.797872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.797882 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:35:19.797924 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.797926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.797951 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.798038 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.798322 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.799085 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.800421 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.801149 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.801233 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.801332 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.801430 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.801468 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260925 02:35:19.801550 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.797420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:40813 (local address 127.0.0.1:53384)
0925 02:35:19.797715 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.797751 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.797805 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.798155 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.798159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.798161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.798233 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.798234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.799178 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.799179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.800329 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.800332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.801246 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.801247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.801327 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.801341 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.801414 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.801478 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":97,"threads_started":1}
I20260925 02:35:19.801991 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.797420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:32973 (local address 127.0.0.1:42890)
0925 02:35:19.797925 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.797938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.798009 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.798240 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.798241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.798243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.798289 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.798290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.799638 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.799639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.800625 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.800627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.801669 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.801670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.801735 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.801741 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.801792 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.801847 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":28,"thread_start_us":94,"threads_started":1}
I20260925 02:35:19.802014 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.797666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.10.13.2:32973)
0925 02:35:19.797866 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.797871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.798006 (+ 135us) server_negotiation.cc:407] Connection header received
0925 02:35:19.798078 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.798080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.798096 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.798168 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.798370 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.799538 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.800706 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.801456 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.801533 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.801627 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.801891 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.801934 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260925 02:35:19.802049 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.797495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.10.13.1:33239)
0925 02:35:19.798169 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.798173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.798182 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:19.798220 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.798222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.798237 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.798309 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.798500 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.799573 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.800793 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.801502 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.801558 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.801631 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.801931 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.801970 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":115,"threads_started":1}
I20260925 02:35:19.802199 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.797403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:33239 (local address 127.0.0.1:52002)
0925 02:35:19.797685 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.797734 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.797800 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.798379 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.798380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.798382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.798414 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.798415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.799660 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.799661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.800713 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.800716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.801668 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.801669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.801735 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.801741 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.801792 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.801837 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260925 02:35:19.802693 29345 tablet_service.cc:1467] Processing CreateTablet for tablet 39048d9a0abc4e7896ca05349e1d5a76 (DEFAULT_TABLE table=test-table [id=81f9f329aa7c4320acb535bdd63e6035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:19.802954 29083 tablet_service.cc:1467] Processing CreateTablet for tablet 39048d9a0abc4e7896ca05349e1d5a76 (DEFAULT_TABLE table=test-table [id=81f9f329aa7c4320acb535bdd63e6035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:19.803129 29345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39048d9a0abc4e7896ca05349e1d5a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:19.803300 29083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39048d9a0abc4e7896ca05349e1d5a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:19.806010 29425 tablet_bootstrap.cc:492] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6: Bootstrap starting.
I20260925 02:35:19.806110 29426 tablet_bootstrap.cc:492] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: Bootstrap starting.
I20260925 02:35:19.806655 29425 tablet_bootstrap.cc:654] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:19.806800 29426 tablet_bootstrap.cc:654] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:19.806710 29214 tablet_service.cc:1467] Processing CreateTablet for tablet 39048d9a0abc4e7896ca05349e1d5a76 (DEFAULT_TABLE table=test-table [id=81f9f329aa7c4320acb535bdd63e6035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:19.807019 29425 log.cc:824] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:19.807101 29214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39048d9a0abc4e7896ca05349e1d5a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:19.807135 29426 log.cc:824] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:19.807832 29425 tablet_bootstrap.cc:492] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6: No bootstrap required, opened a new log
I20260925 02:35:19.807909 29425 ts_tablet_manager.cc:1397] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:35:19.810588 29425 raft_consensus.cc:348] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.810772 29425 raft_consensus.cc:374] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:19.810812 29425 raft_consensus.cc:729] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fdb1253638343be82436b92ab480aa6, State: Initialized, Role: FOLLOWER
I20260925 02:35:19.810902 29429 tablet_bootstrap.cc:492] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Bootstrap starting.
I20260925 02:35:19.810967 29425 consensus_queue.cc:260] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.811260 29425 ts_tablet_manager.cc:1428] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 02:35:19.811431 29149 heartbeater.cc:503] Master 127.10.13.62:32787 was elected leader, sending a full tablet report...
I20260925 02:35:19.811607 29429 tablet_bootstrap.cc:654] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:19.811790 29426 tablet_bootstrap.cc:492] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: No bootstrap required, opened a new log
I20260925 02:35:19.811866 29426 ts_tablet_manager.cc:1397] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 02:35:19.811991 29429 log.cc:824] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:19.812695 29429 tablet_bootstrap.cc:492] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: No bootstrap required, opened a new log
I20260925 02:35:19.812749 29429 ts_tablet_manager.cc:1397] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:35:19.814215 29426 raft_consensus.cc:348] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.814376 29426 raft_consensus.cc:374] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:19.814425 29426 raft_consensus.cc:729] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9a9807aebf4cd7bd6279324a027bac, State: Initialized, Role: FOLLOWER
I20260925 02:35:19.814538 29426 consensus_queue.cc:260] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.814836 29426 ts_tablet_manager.cc:1428] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:19.814925 29411 heartbeater.cc:503] Master 127.10.13.62:32787 was elected leader, sending a full tablet report...
I20260925 02:35:19.814906 29429 raft_consensus.cc:348] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.815037 29429 raft_consensus.cc:374] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:19.815059 29429 raft_consensus.cc:729] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2665aa6704894b51a31d7ac422462b95, State: Initialized, Role: FOLLOWER
I20260925 02:35:19.815191 29429 consensus_queue.cc:260] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.815457 29429 ts_tablet_manager.cc:1428] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:19.815486 29280 heartbeater.cc:503] Master 127.10.13.62:32787 was elected leader, sending a full tablet report...
W20260925 02:35:19.905745 29281 tablet.cc:2369] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:35:19.917680 29430 raft_consensus.cc:482] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:19.917860 29430 raft_consensus.cc:504] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.918251 29430 leader_election.cc:288] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d9a9807aebf4cd7bd6279324a027bac (127.10.13.3:40813), 2665aa6704894b51a31d7ac422462b95 (127.10.13.2:32973)
I20260925 02:35:19.921300 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.918650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:42171 (local address 127.10.13.3:40813)
0925 02:35:19.918740 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.918742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.918752 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:35:19.918877 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.918878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.918888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.918960 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.919123 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.919591 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.920539 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.920925 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.921009 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.921021 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.921180 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.921226 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:35:19.921339 29434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.918462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:40813 (local address 127.10.13.1:42171)
0925 02:35:19.918652 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.918661 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.918705 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.918964 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.918965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.918967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.919050 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.919052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.919680 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.919681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.920442 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.920444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.921023 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.921024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.921104 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.921107 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.921174 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.921215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":21,"thread_start_us":106,"threads_started":1}
I20260925 02:35:19.921366 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.918570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:33889 (local address 127.10.13.2:32973)
0925 02:35:19.918634 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.918636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.918648 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:35:19.918680 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.918681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.918691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.918859 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:19.919021 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.919516 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.920455 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.920857 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.921031 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.921044 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.921222 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.921289 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:35:19.921579 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.918462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:32973 (local address 127.10.13.1:33889)
0925 02:35:19.918528 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.918541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:19.918587 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:19.918866 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:19.918867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:19.918869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:19.918944 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.918946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.919604 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.919605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.920357 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:19.920359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.921032 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.921033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:19.921124 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.921127 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:19.921213 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:35:19.921261 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:35:19.921682 29365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39048d9a0abc4e7896ca05349e1d5a76" candidate_uuid: "0fdb1253638343be82436b92ab480aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9a9807aebf4cd7bd6279324a027bac" is_pre_election: true
I20260925 02:35:19.921866 29365 raft_consensus.cc:2454] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fdb1253638343be82436b92ab480aa6 in term 0.
I20260925 02:35:19.921949 29234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39048d9a0abc4e7896ca05349e1d5a76" candidate_uuid: "0fdb1253638343be82436b92ab480aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2665aa6704894b51a31d7ac422462b95" is_pre_election: true
I20260925 02:35:19.922114 29234 raft_consensus.cc:2454] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fdb1253638343be82436b92ab480aa6 in term 0.
I20260925 02:35:19.922152 29037 leader_election.cc:302] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fdb1253638343be82436b92ab480aa6, 5d9a9807aebf4cd7bd6279324a027bac; no voters:
I20260925 02:35:19.922334 29430 raft_consensus.cc:2781] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:35:19.922429 29430 raft_consensus.cc:482] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:35:19.922463 29430 raft_consensus.cc:3037] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:19.923247 29430 raft_consensus.cc:504] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.923393 29430 leader_election.cc:288] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [CANDIDATE]: Term 1 election: Requested vote from peers 5d9a9807aebf4cd7bd6279324a027bac (127.10.13.3:40813), 2665aa6704894b51a31d7ac422462b95 (127.10.13.2:32973)
I20260925 02:35:19.923532 29234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39048d9a0abc4e7896ca05349e1d5a76" candidate_uuid: "0fdb1253638343be82436b92ab480aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2665aa6704894b51a31d7ac422462b95"
I20260925 02:35:19.923606 29234 raft_consensus.cc:3037] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:19.923631 29365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39048d9a0abc4e7896ca05349e1d5a76" candidate_uuid: "0fdb1253638343be82436b92ab480aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9a9807aebf4cd7bd6279324a027bac"
I20260925 02:35:19.923717 29365 raft_consensus.cc:3037] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:19.924376 29234 raft_consensus.cc:2454] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fdb1253638343be82436b92ab480aa6 in term 1.
I20260925 02:35:19.924548 29365 raft_consensus.cc:2454] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fdb1253638343be82436b92ab480aa6 in term 1.
I20260925 02:35:19.924554 29039 leader_election.cc:302] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fdb1253638343be82436b92ab480aa6, 2665aa6704894b51a31d7ac422462b95; no voters:
I20260925 02:35:19.924751 29430 raft_consensus.cc:2781] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:19.924937 29430 raft_consensus.cc:686] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 0fdb1253638343be82436b92ab480aa6, State: Running, Role: LEADER
I20260925 02:35:19.925153 29430 consensus_queue.cc:237] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } }
I20260925 02:35:19.926095 28960 catalog_manager.cc:5495] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fdb1253638343be82436b92ab480aa6 (127.10.13.1). New cstate: current_term: 1 leader_uuid: "0fdb1253638343be82436b92ab480aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdb1253638343be82436b92ab480aa6" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 33239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 } health_report { overall_health: UNKNOWN } } }
I20260925 02:35:19.947439 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.944995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.10.13.1:33239)
0925 02:35:19.945069 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:19.945071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:19.945078 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:35:19.945116 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:19.945116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:19.945138 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:19.945207 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:19.945326 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.945783 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.946516 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:19.946698 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:19.946764 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:19.946968 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:19.947058 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:19.947166 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:19.947342 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:35:19.947355 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:35:19.949251 29234 raft_consensus.cc:1261] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 1 FOLLOWER]: Refusing update from remote peer 0fdb1253638343be82436b92ab480aa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:19.949292 29365 raft_consensus.cc:1261] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 1 FOLLOWER]: Refusing update from remote peer 0fdb1253638343be82436b92ab480aa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:19.949671 29435 consensus_queue.cc:1035] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9a9807aebf4cd7bd6279324a027bac" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:19.949810 29430 consensus_queue.cc:1035] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2665aa6704894b51a31d7ac422462b95" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:19.953131 29438 mvcc.cc:205] Tried to move back new op lower bound from 7333084036909387776 to 7333084036814303232. Current Snapshot: MvccSnapshot[applied={T|T < 7333084036909387776}]
I20260925 02:35:19.953792 29440 mvcc.cc:205] Tried to move back new op lower bound from 7333084036909387776 to 7333084036814303232. Current Snapshot: MvccSnapshot[applied={T|T < 7333084036909387776}]
I20260925 02:35:19.955476 28961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:19.955582 28961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:19.956393 28961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00a36dadaa1d451b9ad4cbdef56750df: Sending DeleteTablet for 3 replicas of tablet 39048d9a0abc4e7896ca05349e1d5a76
I20260925 02:35:19.956719 29083 tablet_service.cc:1514] Processing DeleteTablet for tablet 39048d9a0abc4e7896ca05349e1d5a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:19 UTC) from {username='slave'} at 127.0.0.1:52002
I20260925 02:35:19.956810 29214 tablet_service.cc:1514] Processing DeleteTablet for tablet 39048d9a0abc4e7896ca05349e1d5a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:19 UTC) from {username='slave'} at 127.0.0.1:42890
I20260925 02:35:19.956967 29345 tablet_service.cc:1514] Processing DeleteTablet for tablet 39048d9a0abc4e7896ca05349e1d5a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:19 UTC) from {username='slave'} at 127.0.0.1:53384
I20260925 02:35:19.957050 29449 tablet_replica.cc:330] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: stopping tablet replica
I20260925 02:35:19.957050 29448 tablet_replica.cc:330] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6: stopping tablet replica
I20260925 02:35:19.957144 29449 raft_consensus.cc:2229] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:19.957146 29448 raft_consensus.cc:2229] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:35:19.957233 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 29021
I20260925 02:35:19.957262 29449 raft_consensus.cc:2258] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:19.957265 29448 raft_consensus.cc:2258] T 39048d9a0abc4e7896ca05349e1d5a76 P 0fdb1253638343be82436b92ab480aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:19.958531 29449 ts_tablet_manager.cc:1905] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:19.958630 29450 tablet_replica.cc:330] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: stopping tablet replica
I20260925 02:35:19.958772 29450 raft_consensus.cc:2229] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:19.958936 29450 raft_consensus.cc:2258] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:19.959389 29450 ts_tablet_manager.cc:1905] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:19.960513 29449 ts_tablet_manager.cc:1918] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:19.960592 29449 log.cc:1196] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/39048d9a0abc4e7896ca05349e1d5a76
I20260925 02:35:19.960952 29449 ts_tablet_manager.cc:1939] T 39048d9a0abc4e7896ca05349e1d5a76 P 2665aa6704894b51a31d7ac422462b95: Deleting consensus metadata
I20260925 02:35:19.961395 29450 ts_tablet_manager.cc:1918] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:19.961412 28948 catalog_manager.cc:4841] TS 2665aa6704894b51a31d7ac422462b95 (127.10.13.2:32973): tablet 39048d9a0abc4e7896ca05349e1d5a76 (table test-table [id=81f9f329aa7c4320acb535bdd63e6035]) successfully deleted
I20260925 02:35:19.961470 29450 log.cc:1196] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/39048d9a0abc4e7896ca05349e1d5a76
I20260925 02:35:19.961702 29450 ts_tablet_manager.cc:1939] T 39048d9a0abc4e7896ca05349e1d5a76 P 5d9a9807aebf4cd7bd6279324a027bac: Deleting consensus metadata
I20260925 02:35:19.962167 28946 catalog_manager.cc:4841] TS 5d9a9807aebf4cd7bd6279324a027bac (127.10.13.3:40813): tablet 39048d9a0abc4e7896ca05349e1d5a76 (table test-table [id=81f9f329aa7c4320acb535bdd63e6035]) successfully deleted
W20260925 02:35:19.964551 28947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:33239 (error 108)
I20260925 02:35:19.965054 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:19.964949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:33239 (local address 127.0.0.1:52016)
0925 02:35:19.964987 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:19.965027 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:33239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 02:35:19.965190 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 29152
W20260925 02:35:19.965200 28947 catalog_manager.cc:4584] TS 0fdb1253638343be82436b92ab480aa6 (127.10.13.1:33239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39048d9a0abc4e7896ca05349e1d5a76: Network error: Client connection negotiation failed: client connection to 127.10.13.1:33239: connect: Connection refused (error 111)
I20260925 02:35:19.970469 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 29283
I20260925 02:35:19.975507 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 28930
2026-09-25T02:35:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (746 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:20 dist-test-slave-q8mg krb5kdc[29454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:20 dist-test-slave-q8mg krb5kdc[29454](info): set up 2 sockets
Sep 25 02:35:20 dist-test-slave-q8mg krb5kdc[29454](info): commencing operation
krb5kdc: starting...
W20260925 02:35:21.023185 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:21.148815 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:43119
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:43119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:21.234864 29467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:21.235067 29467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:21.235090 29467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:21.236182 29467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:21.236220 29467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:21.236235 29467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:21.236248 29467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:21.236259 29467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:21.237777 29467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:43119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:43119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:21.237967 29467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:21.238139 29467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:21.243093 29473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:21.243116 29472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:21.243098 29475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:21.243134 29467 server_base.cc:1034] running on GCE node
I20260925 02:35:21.243615 29467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:21.243969 29467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:21.245116 29467 hybrid_clock.cc:648] HybridClock initialized: now 1790303721245096 us; error 32 us; skew 500 ppm
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:21.252689 29467 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:21.253320 29467 webserver.cc:460] Webserver started at http://127.10.13.62:41489/ using document root <none> and password file <none>
I20260925 02:35:21.253584 29467 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:21.253659 29467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:21.253810 29467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:21.254916 29467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "62acccc06ec943af80257d4330bffbfe"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "cee60db22337a23bb1d3f7531870648a"
server_key_iv: "40c78f757bcfc8652554c15929fdd7bd"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.255286 29467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "62acccc06ec943af80257d4330bffbfe"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "cee60db22337a23bb1d3f7531870648a"
server_key_iv: "40c78f757bcfc8652554c15929fdd7bd"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.256683 29467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:21.257624 29482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.257843 29467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:21.257962 29467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "62acccc06ec943af80257d4330bffbfe"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "cee60db22337a23bb1d3f7531870648a"
server_key_iv: "40c78f757bcfc8652554c15929fdd7bd"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.258042 29467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:21.267694 29467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:21.269676 29467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:21.269855 29467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:21.274151 29467 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:43119
I20260925 02:35:21.274201 29534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:43119 every 8 connection(s)
I20260925 02:35:21.274588 29467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:21.275528 29535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:21.277961 29535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe: Bootstrap starting.
I20260925 02:35:21.278570 29535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:21.278868 29535 log.cc:824] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:21.279629 29535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe: No bootstrap required, opened a new log
I20260925 02:35:21.281347 29535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acccc06ec943af80257d4330bffbfe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43119 } }
I20260925 02:35:21.281491 29535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:21.281515 29535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62acccc06ec943af80257d4330bffbfe, State: Initialized, Role: FOLLOWER
I20260925 02:35:21.281697 29535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acccc06ec943af80257d4330bffbfe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43119 } }
I20260925 02:35:21.281772 29535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:21.281795 29535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:21.281826 29535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:21.282384 29535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acccc06ec943af80257d4330bffbfe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43119 } }
I20260925 02:35:21.282480 29535 leader_election.cc:302] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62acccc06ec943af80257d4330bffbfe; no voters:
I20260925 02:35:21.282696 29535 leader_election.cc:288] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:21.282747 29538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:21.282982 29538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [term 1 LEADER]: Becoming Leader. State: Replica: 62acccc06ec943af80257d4330bffbfe, State: Running, Role: LEADER
I20260925 02:35:21.282987 29535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:21.283088 29538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acccc06ec943af80257d4330bffbfe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43119 } }
I20260925 02:35:21.283471 29540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62acccc06ec943af80257d4330bffbfe. Latest consensus state: current_term: 1 leader_uuid: "62acccc06ec943af80257d4330bffbfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acccc06ec943af80257d4330bffbfe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43119 } } }
I20260925 02:35:21.283560 29540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:21.283618 29539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62acccc06ec943af80257d4330bffbfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acccc06ec943af80257d4330bffbfe" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 43119 } } }
I20260925 02:35:21.283691 29539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:21.284193 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 29467
I20260925 02:35:21.284324 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:21.284545 10292 external_mini_cluster.cc:1428] Setting key e4cc2798091d88119bf9dd79325a4ea0
W20260925 02:35:21.285121 29554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:21.285213 29554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:21.285341 29546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:21.285557 29546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:21.287048 29546 catalog_manager.cc:1347] Generated new cluster ID: a9a48a85b9834d1990bf319af4d88c90
I20260925 02:35:21.287101 29546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:21.300541 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.285076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.10.13.62:43119)
0925 02:35:21.289231 (+ 4155us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:21.289241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:21.289273 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:35:21.289388 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:21.289395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:21.289464 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:21.289583 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:21.290386 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.291850 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.293070 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.293724 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.293826 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.296749 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:21.296754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:21.296755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:21.296768 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:21.298844 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.299227 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.299231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.299232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.299355 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.299562 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.299563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.299564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.299909 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:21.300066 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:21.300140 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:35:21.300281 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4102,"thread_start_us":70,"threads_started":1}
I20260925 02:35:21.306293 29546 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:21.306818 29546 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:21.313583 29546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe: Generated new TSK 0
I20260925 02:35:21.313850 29546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.13.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
Entry for principal kudu/127.10.13.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab.
I20260925 02:35:21.344609 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43119
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:21.432386 29561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:21.432602 29561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:21.432628 29561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:21.433987 29561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:21.434039 29561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:21.434068 29561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:35:21.435590 29561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.1.keytab
--principal=kudu/127.10.13.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:21.435871 29561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:21.436172 29561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:21.441721 29566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:21.441766 29569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:21.441704 29567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:21.441818 29561 server_base.cc:1034] running on GCE node
I20260925 02:35:21.442219 29561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:21.442499 29561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:21.443708 29561 hybrid_clock.cc:648] HybridClock initialized: now 1790303721443682 us; error 57 us; skew 500 ppm
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:21.446472 29561 init.cc:377] Logged in from keytab as kudu/127.10.13.1@KRBTEST.COM (short username kudu)
I20260925 02:35:21.447036 29561 webserver.cc:460] Webserver started at http://127.10.13.1:33807/ using document root <none> and password file <none>
I20260925 02:35:21.447278 29561 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:21.447347 29561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:21.447510 29561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:21.448607 29561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "2cb28339876d4d2c9b44560c7b89e908"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "0fbe909ea107b2d42fa37006f0af6af1"
server_key_iv: "3b68939884611716b46e97636d34b717"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.449043 29561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "2cb28339876d4d2c9b44560c7b89e908"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "0fbe909ea107b2d42fa37006f0af6af1"
server_key_iv: "3b68939884611716b46e97636d34b717"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.450611 29561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:35:21.451519 29576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.451925 29561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:21.452045 29561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "2cb28339876d4d2c9b44560c7b89e908"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "0fbe909ea107b2d42fa37006f0af6af1"
server_key_iv: "3b68939884611716b46e97636d34b717"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.452126 29561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:21.464043 29561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:21.467046 29561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:21.467228 29561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:21.467489 29561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:21.467825 29561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:21.467880 29561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.467914 29561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:21.467941 29561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.474311 29561 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:40309
I20260925 02:35:21.474370 29689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:40309 every 8 connection(s)
I20260925 02:35:21.474684 29561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:35:21.480120 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 29561
I20260925 02:35:21.480219 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:35:21.480547 10292 external_mini_cluster.cc:1428] Setting key 2594bab48b2d98fe05895a2cda8540db
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.1@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:21.485144 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.475415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:45251 (local address 127.10.13.62:43119)
0925 02:35:21.475491 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:21.475494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:21.475702 (+ 208us) server_negotiation.cc:407] Connection header received
0925 02:35:21.476206 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:21.476207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:21.476217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:21.476271 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:21.477355 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.477903 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.478905 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.479114 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.479192 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.481759 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:21.481761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:21.481761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:21.481770 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:21.483466 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.484000 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.484001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.484001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.484088 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.484393 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.484394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.484394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.484505 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:21.484560 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:21.484995 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 02:35:21.485077 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:35:21.485431 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.475391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43119 (local address 127.10.13.1:45251)
0925 02:35:21.475585 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:21.475627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:21.476113 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:21.476353 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:21.476356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:21.477037 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:21.477243 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.477249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.478001 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.478002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.478820 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.478822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.479217 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.479217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.479335 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.479902 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:21.479911 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:21.481652 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:21.483566 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:21.483574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:21.483578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:21.483920 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:21.484166 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:21.484167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:21.484167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:21.484325 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:21.484574 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:21.484582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:21.484756 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 02:35:21.484952 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 02:35:21.485148 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260925 02:35:21.485888 29690 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43119
I20260925 02:35:21.486009 29690 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:21.486205 29690 heartbeater.cc:511] Master 127.10.13.62:43119 requested a full tablet report, sending...
I20260925 02:35:21.486804 29499 ts_manager.cc:194] Registered new tserver with Master: 2cb28339876d4d2c9b44560c7b89e908 (127.10.13.1:40309)
I20260925 02:35:21.487434 29499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.1@KRBTEST.COM'} at 127.10.13.1:45251
WARNING: no policy specified for kudu/127.10.13.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
Entry for principal kudu/127.10.13.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab.
I20260925 02:35:21.508111 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43119
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:21.593374 29695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:21.593591 29695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:21.593617 29695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:21.594775 29695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:21.594813 29695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:21.594846 29695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:35:21.596343 29695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.2.keytab
--principal=kudu/127.10.13.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:21.596618 29695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:21.596899 29695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:21.602208 29700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:21.602283 29701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:21.602455 29703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:21.602455 29695 server_base.cc:1034] running on GCE node
I20260925 02:35:21.602710 29695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:21.602959 29695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:21.604126 29695 hybrid_clock.cc:648] HybridClock initialized: now 1790303721604124 us; error 38 us; skew 500 ppm
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:21.606650 29695 init.cc:377] Logged in from keytab as kudu/127.10.13.2@KRBTEST.COM (short username kudu)
I20260925 02:35:21.607160 29695 webserver.cc:460] Webserver started at http://127.10.13.2:37775/ using document root <none> and password file <none>
I20260925 02:35:21.607400 29695 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:21.607471 29695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:21.607627 29695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:21.608732 29695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "aa7f17db9c68493a8494c45054cfa516"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "83fb4ab448c0f8b785602ed1a791fd9b"
server_key_iv: "16a29f3f82dbfb478c6d10ac75e1e2ae"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.609155 29695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "aa7f17db9c68493a8494c45054cfa516"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "83fb4ab448c0f8b785602ed1a791fd9b"
server_key_iv: "16a29f3f82dbfb478c6d10ac75e1e2ae"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.610574 29695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:21.611343 29710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.611552 29695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:21.611665 29695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "aa7f17db9c68493a8494c45054cfa516"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "83fb4ab448c0f8b785602ed1a791fd9b"
server_key_iv: "16a29f3f82dbfb478c6d10ac75e1e2ae"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.611742 29695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:21.623668 29695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:21.625681 29695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:21.625864 29695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:21.626111 29695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:21.626441 29695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:21.626495 29695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.626537 29695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:21.626565 29695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.633112 29695 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:42165
I20260925 02:35:21.633160 29823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:42165 every 8 connection(s)
I20260925 02:35:21.633500 29695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.2@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:21.642977 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.634238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:38627 (local address 127.10.13.62:43119)
0925 02:35:21.634335 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:21.634337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:21.634550 (+ 213us) server_negotiation.cc:407] Connection header received
0925 02:35:21.635056 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:21.635057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:21.635068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:21.635129 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:21.636101 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.636601 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.637668 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.637878 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.637934 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.640252 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:21.640254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:21.640254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:21.640261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:21.641794 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.642149 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.642149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.642150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.642199 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.642411 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.642412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.642412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.642514 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:21.642563 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:21.642841 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 02:35:21.642910 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:35:21.643172 29826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.634227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43119 (local address 127.10.13.2:38627)
0925 02:35:21.634415 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:21.634460 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:21.634962 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:21.635184 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:21.635188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:21.635796 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:21.635998 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.636003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.636687 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.636688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.637594 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.637597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.637969 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.637970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.638104 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.638653 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:21.638662 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:21.640161 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:21.641864 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:21.641870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:21.641872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:21.642083 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:21.642256 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:21.642256 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:21.642257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:21.642357 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:21.642568 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:21.642573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:21.642690 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 02:35:21.642774 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:35:21.642930 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260925 02:35:21.643427 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 29695
I20260925 02:35:21.643550 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:35:21.643720 29824 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43119
I20260925 02:35:21.643793 10292 external_mini_cluster.cc:1428] Setting key a9d1609e62ead29daf4a04fb8dbbd7b1
I20260925 02:35:21.643826 29824 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:21.644021 29824 heartbeater.cc:511] Master 127.10.13.62:43119 requested a full tablet report, sending...
I20260925 02:35:21.644500 29499 ts_manager.cc:194] Registered new tserver with Master: aa7f17db9c68493a8494c45054cfa516 (127.10.13.2:42165)
I20260925 02:35:21.644934 29499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.2@KRBTEST.COM'} at 127.10.13.2:38627
WARNING: no policy specified for kudu/127.10.13.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
Entry for principal kudu/127.10.13.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab.
I20260925 02:35:21.669306 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43119
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:21.752279 29829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:21.752482 29829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:21.752506 29829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:21.753767 29829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:21.753810 29829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:21.753837 29829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:35:21.755353 29829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.3.keytab
--principal=kudu/127.10.13.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:43119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:21.755616 29829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:21.755831 29829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:21.760787 29834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:21.760854 29837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:21.760986 29829 server_base.cc:1034] running on GCE node
W20260925 02:35:21.760794 29835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:21.761351 29829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:21.761689 29829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:21.762871 29829 hybrid_clock.cc:648] HybridClock initialized: now 1790303721762866 us; error 35 us; skew 500 ppm
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:21.765439 29829 init.cc:377] Logged in from keytab as kudu/127.10.13.3@KRBTEST.COM (short username kudu)
I20260925 02:35:21.765888 29829 webserver.cc:460] Webserver started at http://127.10.13.3:37871/ using document root <none> and password file <none>
I20260925 02:35:21.766155 29829 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:21.766227 29829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:21.766383 29829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:21.767453 29829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "b79b7641568741e190a6d4d995150651"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "3471807cdd6d6893f2ffa2f4191ffdb2"
server_key_iv: "74392807ff9f2eec136ec94bc0ec1a7f"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.767817 29829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "b79b7641568741e190a6d4d995150651"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "3471807cdd6d6893f2ffa2f4191ffdb2"
server_key_iv: "74392807ff9f2eec136ec94bc0ec1a7f"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.769227 29829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:21.770037 29844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.770228 29829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:35:21.770342 29829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "b79b7641568741e190a6d4d995150651"
format_stamp: "Formatted at 2026-09-25 02:35:21 on dist-test-slave-q8mg"
server_key: "3471807cdd6d6893f2ffa2f4191ffdb2"
server_key_iv: "74392807ff9f2eec136ec94bc0ec1a7f"
server_key_version: "encryptionkey@0"
I20260925 02:35:21.770416 29829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:21.806252 29829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:21.808193 29829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:21.808377 29829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:21.808643 29829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:21.809038 29829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:21.809093 29829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.809135 29829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:21.809165 29829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:21.815637 29829 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:43091
I20260925 02:35:21.815680 29958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:43091 every 8 connection(s)
I20260925 02:35:21.816133 29829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:35:21 dist-test-slave-q8mg krb5kdc[29454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303721, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.3@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:21.825034 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 29829
I20260925 02:35:21.825150 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:35:21.825337 10292 external_mini_cluster.cc:1428] Setting key 1e5baa56f74742b9d8d588de3335d798
I20260925 02:35:21.825381 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.817028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:44099 (local address 127.10.13.62:43119)
0925 02:35:21.817089 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:21.817091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:21.817334 (+ 243us) server_negotiation.cc:407] Connection header received
0925 02:35:21.817817 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:21.817818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:21.817828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:21.817896 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:21.818841 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.819330 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.820298 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.820483 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.820534 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.822727 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:21.822729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:21.822729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:21.822739 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:21.823952 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.824356 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.824357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.824358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.824420 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.824675 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.824676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.824676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.824803 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:21.824901 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:21.825205 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:35:21.825290 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:35:21.825654 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.817031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:43119 (local address 127.10.13.3:44099)
0925 02:35:21.817189 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:21.817233 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:21.817736 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:21.817955 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:21.817959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:21.818552 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:21.818760 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.818765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.819417 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.819417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.820223 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.820224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.820559 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.820561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.820666 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.821197 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:21.821205 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:21.822618 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:21.824040 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:21.824047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:21.824050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:21.824275 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:21.824490 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:21.824491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:21.824491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:21.824604 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:21.824897 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:21.824905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:21.825057 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 02:35:21.825165 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:35:21.825327 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 02:35:21.826174 29959 heartbeater.cc:347] Connected to a master server at 127.10.13.62:43119
I20260925 02:35:21.826280 29959 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:21.826498 29959 heartbeater.cc:511] Master 127.10.13.62:43119 requested a full tablet report, sending...
I20260925 02:35:21.826857 29499 ts_manager.cc:194] Registered new tserver with Master: b79b7641568741e190a6d4d995150651 (127.10.13.3:43091)
I20260925 02:35:21.827209 29499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.13.3@KRBTEST.COM'} at 127.10.13.3:44099
I20260925 02:35:21.837397 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:35:21.846127 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.838859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.10.13.62:43119)
0925 02:35:21.838926 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:21.838928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:21.839025 (+ 97us) server_negotiation.cc:407] Connection header received
0925 02:35:21.839084 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:21.839084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:21.839094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:21.839141 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:21.839886 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.840746 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.841547 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.841758 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.841888 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.843146 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:21.843148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:21.843148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:21.843156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:21.844588 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.845135 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.845137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.845137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.845208 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:21.845462 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:21.845464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:21.845464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:21.845640 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:21.845755 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:21.845920 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 02:35:21.846009 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:35:21.847851 29499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:35:21.849740 29499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:35:21.857172 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.852903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.10.13.1:40309)
0925 02:35:21.853190 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:21.853194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:21.853205 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:35:21.853245 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:21.853248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:21.853265 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:21.853337 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:21.853573 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.854826 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.855948 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.856676 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.856759 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.856836 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:21.857036 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 02:35:21.857076 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":119,"threads_started":1}
I20260925 02:35:21.857191 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.852286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:42165 (local address 127.0.0.1:38258)
0925 02:35:21.852460 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:21.852527 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:21.852603 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:21.853129 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:21.853133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:21.853136 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:21.853221 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.853223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.854448 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.854449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.855908 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.855912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.856783 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.856784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.856897 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.856923 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:35:21.856977 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:35:21.857058 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260925 02:35:21.857379 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.852562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.10.13.2:42165)
0925 02:35:21.852948 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:21.852953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:21.852966 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:35:21.853007 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:21.853009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:21.853029 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:21.853120 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:21.853323 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.854337 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.856011 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.856687 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.857141 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.857209 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:21.857235 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 02:35:21.857277 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":100,"threads_started":1}
I20260925 02:35:21.857409 29973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.852366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:43091 (local address 127.0.0.1:41480)
0925 02:35:21.853236 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:21.853246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:21.853310 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:21.853553 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:21.853554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:21.853556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:21.853618 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.853620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.854894 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.854895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.856218 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.856220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.857122 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.857123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.857194 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.857199 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:35:21.857269 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:35:21.857311 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":101,"thread_start_us":96,"threads_started":1}
I20260925 02:35:21.857753 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.852614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.10.13.3:43091)
0925 02:35:21.852813 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:21.852819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:21.853373 (+ 554us) server_negotiation.cc:407] Connection header received
0925 02:35:21.853434 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:21.853437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:21.853453 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:21.853533 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:21.853712 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.854768 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.856307 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.856957 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.857025 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.857084 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:21.857620 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 02:35:21.857666 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260925 02:35:21.858217 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:21.852366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:40309 (local address 127.0.0.1:49546)
0925 02:35:21.852530 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:21.852541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:21.852603 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:21.853421 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:21.853422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:21.853424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:21.853481 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.853482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.854909 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.854910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.855851 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:21.855854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:21.856780 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:21.856781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:21.856896 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:21.856922 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:35:21.857013 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:35:21.857061 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":34,"thread_start_us":98,"threads_started":1}
I20260925 02:35:21.858456 29748 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5 (DEFAULT_TABLE table=test-table [id=0033ddb5a7294efda016106a963b25dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:21.858860 29748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:21.858985 29624 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5 (DEFAULT_TABLE table=test-table [id=0033ddb5a7294efda016106a963b25dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:21.859335 29624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:21.862041 29977 tablet_bootstrap.cc:492] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Bootstrap starting.
I20260925 02:35:21.862504 29978 tablet_bootstrap.cc:492] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908: Bootstrap starting.
I20260925 02:35:21.862497 29892 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5 (DEFAULT_TABLE table=test-table [id=0033ddb5a7294efda016106a963b25dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:21.862746 29977 tablet_bootstrap.cc:654] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:21.862829 29892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:21.863235 29978 tablet_bootstrap.cc:654] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:21.863354 29977 log.cc:824] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:21.863921 29978 log.cc:824] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:21.864257 29977 tablet_bootstrap.cc:492] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: No bootstrap required, opened a new log
I20260925 02:35:21.864343 29977 ts_tablet_manager.cc:1397] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:35:21.864707 29978 tablet_bootstrap.cc:492] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908: No bootstrap required, opened a new log
I20260925 02:35:21.864802 29978 ts_tablet_manager.cc:1397] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:35:21.866346 29981 tablet_bootstrap.cc:492] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Bootstrap starting.
I20260925 02:35:21.866371 29978 raft_consensus.cc:348] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:21.866580 29978 raft_consensus.cc:374] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:21.866623 29978 raft_consensus.cc:729] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb28339876d4d2c9b44560c7b89e908, State: Initialized, Role: FOLLOWER
I20260925 02:35:21.866463 29977 raft_consensus.cc:348] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:21.866672 29977 raft_consensus.cc:374] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:21.866710 29977 raft_consensus.cc:729] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7f17db9c68493a8494c45054cfa516, State: Initialized, Role: FOLLOWER
I20260925 02:35:21.866791 29978 consensus_queue.cc:260] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:21.866834 29977 consensus_queue.cc:260] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:21.867122 29977 ts_tablet_manager.cc:1428] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:21.867133 29978 ts_tablet_manager.cc:1428] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 02:35:21.867188 29981 tablet_bootstrap.cc:654] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:21.867254 29824 heartbeater.cc:503] Master 127.10.13.62:43119 was elected leader, sending a full tablet report...
I20260925 02:35:21.867278 29690 heartbeater.cc:503] Master 127.10.13.62:43119 was elected leader, sending a full tablet report...
I20260925 02:35:21.867841 29981 log.cc:824] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:21.869024 29981 tablet_bootstrap.cc:492] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: No bootstrap required, opened a new log
I20260925 02:35:21.869086 29981 ts_tablet_manager.cc:1397] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:35:21.870648 29981 raft_consensus.cc:348] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:21.870811 29981 raft_consensus.cc:374] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:21.870837 29981 raft_consensus.cc:729] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79b7641568741e190a6d4d995150651, State: Initialized, Role: FOLLOWER
I20260925 02:35:21.870972 29981 consensus_queue.cc:260] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:21.871258 29981 ts_tablet_manager.cc:1428] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:35:21.871300 29959 heartbeater.cc:503] Master 127.10.13.62:43119 was elected leader, sending a full tablet report...
W20260925 02:35:21.884120 29825 tablet.cc:2369] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:35:21.975423 29691 tablet.cc:2369] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:35:22.066788 29960 tablet.cc:2369] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:35:22.074681 29983 raft_consensus.cc:482] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:22.074827 29983 raft_consensus.cc:504] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:22.075155 29983 leader_election.cc:288] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b79b7641568741e190a6d4d995150651 (127.10.13.3:43091), 2cb28339876d4d2c9b44560c7b89e908 (127.10.13.1:40309)
I20260925 02:35:22.078096 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.075468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:51545 (local address 127.10.13.1:40309)
0925 02:35:22.075528 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.075531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.075542 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:35:22.075620 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.075621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.075634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.075709 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:22.075924 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.076404 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.077357 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.077734 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.077814 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.077826 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.077985 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.078025 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:35:22.078096 29826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.075383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:40309 (local address 127.10.13.2:51545)
0925 02:35:22.075448 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.075462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.075538 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.075714 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.075715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.075717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:22.075844 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.075847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.076491 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.076492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.077263 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.077265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.077827 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.077828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.077911 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.077914 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.077979 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.078021 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 02:35:22.078446 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.075464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:50155 (local address 127.10.13.3:43091)
0925 02:35:22.075519 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.075521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.075827 (+ 306us) server_negotiation.cc:407] Connection header received
0925 02:35:22.075897 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.075898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.075911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.075961 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:22.076136 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.076608 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.077639 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.078041 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.078146 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.078157 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.078335 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.078380 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:35:22.078445 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.075383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:43091 (local address 127.10.13.2:50155)
0925 02:35:22.075744 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.075754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.075841 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.075977 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.075978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.075979 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:22.076056 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.076057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.076700 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.076701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.077541 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.077543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.078152 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.078153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.078235 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.078241 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.078321 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.078360 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":168,"thread_start_us":134,"threads_started":1}
I20260925 02:35:22.078467 29644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6e834ebb6c4c17bc9f1e9d9315a7c5" candidate_uuid: "aa7f17db9c68493a8494c45054cfa516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb28339876d4d2c9b44560c7b89e908" is_pre_election: true
I20260925 02:35:22.078663 29644 raft_consensus.cc:2454] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7f17db9c68493a8494c45054cfa516 in term 0.
I20260925 02:35:22.078832 29912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6e834ebb6c4c17bc9f1e9d9315a7c5" candidate_uuid: "aa7f17db9c68493a8494c45054cfa516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79b7641568741e190a6d4d995150651" is_pre_election: true
I20260925 02:35:22.078927 29714 leader_election.cc:302] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb28339876d4d2c9b44560c7b89e908, aa7f17db9c68493a8494c45054cfa516; no voters:
I20260925 02:35:22.079003 29912 raft_consensus.cc:2454] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7f17db9c68493a8494c45054cfa516 in term 0.
I20260925 02:35:22.079236 29983 raft_consensus.cc:2781] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:35:22.079334 29983 raft_consensus.cc:482] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:35:22.079371 29983 raft_consensus.cc:3037] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:22.080077 29983 raft_consensus.cc:504] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:22.080228 29983 leader_election.cc:288] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [CANDIDATE]: Term 1 election: Requested vote from peers b79b7641568741e190a6d4d995150651 (127.10.13.3:43091), 2cb28339876d4d2c9b44560c7b89e908 (127.10.13.1:40309)
I20260925 02:35:22.080399 29912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6e834ebb6c4c17bc9f1e9d9315a7c5" candidate_uuid: "aa7f17db9c68493a8494c45054cfa516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79b7641568741e190a6d4d995150651"
I20260925 02:35:22.080461 29912 raft_consensus.cc:3037] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:22.081001 29644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6e834ebb6c4c17bc9f1e9d9315a7c5" candidate_uuid: "aa7f17db9c68493a8494c45054cfa516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb28339876d4d2c9b44560c7b89e908"
I20260925 02:35:22.081103 29644 raft_consensus.cc:3037] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:22.081244 29912 raft_consensus.cc:2454] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7f17db9c68493a8494c45054cfa516 in term 1.
I20260925 02:35:22.081414 29713 leader_election.cc:302] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa7f17db9c68493a8494c45054cfa516, b79b7641568741e190a6d4d995150651; no voters:
I20260925 02:35:22.081514 29983 raft_consensus.cc:2781] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:22.081663 29983 raft_consensus.cc:686] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 1 LEADER]: Becoming Leader. State: Replica: aa7f17db9c68493a8494c45054cfa516, State: Running, Role: LEADER
I20260925 02:35:22.081785 29983 consensus_queue.cc:237] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } }
I20260925 02:35:22.082070 29644 raft_consensus.cc:2454] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7f17db9c68493a8494c45054cfa516 in term 1.
I20260925 02:35:22.082655 29497 catalog_manager.cc:5495] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa7f17db9c68493a8494c45054cfa516 (127.10.13.2). New cstate: current_term: 1 leader_uuid: "aa7f17db9c68493a8494c45054cfa516" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7f17db9c68493a8494c45054cfa516" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 42165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 } health_report { overall_health: UNKNOWN } } }
I20260925 02:35:22.140286 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.137641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.10.13.2:42165)
0925 02:35:22.137719 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.137721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.137730 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:22.137773 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.137774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.137811 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.137886 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:22.138061 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.138605 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.139422 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.139640 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.139715 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.139906 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:22.140024 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:22.140124 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.140144 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.140167 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:35:22.142225 29912 raft_consensus.cc:1261] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 1 FOLLOWER]: Refusing update from remote peer aa7f17db9c68493a8494c45054cfa516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:22.142251 29644 raft_consensus.cc:1261] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P 2cb28339876d4d2c9b44560c7b89e908 [term 1 FOLLOWER]: Refusing update from remote peer aa7f17db9c68493a8494c45054cfa516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:22.142637 29987 consensus_queue.cc:1035] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb28339876d4d2c9b44560c7b89e908" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 40309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:22.142772 29983 consensus_queue.cc:1035] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79b7641568741e190a6d4d995150651" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:22.145694 29991 mvcc.cc:205] Tried to move back new op lower bound from 7333084045891551232 to 7333084045647527936. Current Snapshot: MvccSnapshot[applied={T|T < 7333084045891551232}]
I20260925 02:35:22.148049 29497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:22.148144 29497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:22.148988 29497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62acccc06ec943af80257d4330bffbfe: Sending DeleteTablet for 3 replicas of tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5
I20260925 02:35:22.149317 29892 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:22 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:41480
I20260925 02:35:22.149346 29748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:22 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:38258
I20260925 02:35:22.149366 29624 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:22 UTC) from {username='kudu', principal='kudu/127.10.13.62@KRBTEST.COM'} at 127.0.0.1:49546
I20260925 02:35:22.149612 30001 tablet_replica.cc:330] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: stopping tablet replica
I20260925 02:35:22.149719 30001 raft_consensus.cc:2229] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:22.149823 30001 raft_consensus.cc:2258] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:22.149976 30000 tablet_replica.cc:330] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: stopping tablet replica
I20260925 02:35:22.150008 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 29561
I20260925 02:35:22.150076 30000 raft_consensus.cc:2229] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:35:22.150187 30000 raft_consensus.cc:2258] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:22.151146 30001 ts_tablet_manager.cc:1905] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:22.151408 30000 ts_tablet_manager.cc:1905] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:22.153116 30001 ts_tablet_manager.cc:1918] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:22.153257 30001 log.cc:1196] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/1d6e834ebb6c4c17bc9f1e9d9315a7c5
I20260925 02:35:22.153440 30000 ts_tablet_manager.cc:1918] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:22.153518 30000 log.cc:1196] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/1d6e834ebb6c4c17bc9f1e9d9315a7c5
I20260925 02:35:22.153746 30001 ts_tablet_manager.cc:1939] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P b79b7641568741e190a6d4d995150651: Deleting consensus metadata
I20260925 02:35:22.153815 30000 ts_tablet_manager.cc:1939] T 1d6e834ebb6c4c17bc9f1e9d9315a7c5 P aa7f17db9c68493a8494c45054cfa516: Deleting consensus metadata
I20260925 02:35:22.154196 29485 catalog_manager.cc:4841] TS b79b7641568741e190a6d4d995150651 (127.10.13.3:43091): tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5 (table test-table [id=0033ddb5a7294efda016106a963b25dd]) successfully deleted
I20260925 02:35:22.154507 29483 catalog_manager.cc:4841] TS aa7f17db9c68493a8494c45054cfa516 (127.10.13.2:42165): tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5 (table test-table [id=0033ddb5a7294efda016106a963b25dd]) successfully deleted
W20260925 02:35:22.161732 29486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:40309 (error 108)
I20260925 02:35:22.161912 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 29695
I20260925 02:35:22.163913 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.163747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:40309 (local address 127.0.0.1:49552)
0925 02:35:22.163822 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.163869 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:35:22.164058 29486 catalog_manager.cc:4584] TS 2cb28339876d4d2c9b44560c7b89e908 (127.10.13.1:40309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d6e834ebb6c4c17bc9f1e9d9315a7c5: Network error: Client connection negotiation failed: client connection to 127.10.13.1:40309: connect: Connection refused (error 111)
I20260925 02:35:22.167116 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 29829
I20260925 02:35:22.171837 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 29467
2026-09-25T02:35:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2197 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T02:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:22Z Disabled control of system clock
I20260925 02:35:22.199944 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:35093
--webserver_interface=127.10.13.62
--webserver_port=0
--builtin_ntp_servers=127.10.13.20:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:35093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:22.281766 30007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:22.281988 30007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:22.282009 30007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:22.283119 30007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:22.283155 30007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:22.283169 30007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:22.283180 30007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:22.283192 30007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:22.284570 30007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:35093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:35093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:22.284793 30007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:22.285108 30007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:22.290023 30013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:22.290050 30007 server_base.cc:1034] running on GCE node
W20260925 02:35:22.290056 30012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:22.290046 30015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:22.290393 30007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:22.290676 30007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:22.291849 30007 hybrid_clock.cc:648] HybridClock initialized: now 1790303722291835 us; error 44 us; skew 500 ppm
I20260925 02:35:22.293147 30007 webserver.cc:460] Webserver started at http://127.10.13.62:33499/ using document root <none> and password file <none>
I20260925 02:35:22.293391 30007 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:22.293458 30007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:22.293608 30007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:22.294750 30007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "e764e9f5afd04036a8c6119c1e1b7df6"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "75966c80df81f88b39732236ec385215"
server_key_iv: "83ab97eb6780d55d8dad37dc1c117e46"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.295126 30007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "e764e9f5afd04036a8c6119c1e1b7df6"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "75966c80df81f88b39732236ec385215"
server_key_iv: "83ab97eb6780d55d8dad37dc1c117e46"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.296506 30007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:22.297420 30022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.297598 30007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:35:22.297714 30007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "e764e9f5afd04036a8c6119c1e1b7df6"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "75966c80df81f88b39732236ec385215"
server_key_iv: "83ab97eb6780d55d8dad37dc1c117e46"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.297817 30007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:22.315171 30007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:22.315515 30007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:22.315701 30007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:22.320048 30007 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:35093
I20260925 02:35:22.320173 30074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:35093 every 8 connection(s)
I20260925 02:35:22.320472 30007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:22.321487 30075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:22.323855 30075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6: Bootstrap starting.
I20260925 02:35:22.324508 30075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:22.324530 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 30007
I20260925 02:35:22.324667 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:22.324844 30075 log.cc:824] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:22.324930 10292 external_mini_cluster.cc:1428] Setting key 5fbc46aaf5abd2a11359081cc612783f
I20260925 02:35:22.325755 30075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6: No bootstrap required, opened a new log
I20260925 02:35:22.328018 30075 raft_consensus.cc:348] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e764e9f5afd04036a8c6119c1e1b7df6" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35093 } }
I20260925 02:35:22.328243 30075 raft_consensus.cc:374] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:22.328286 30075 raft_consensus.cc:729] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e764e9f5afd04036a8c6119c1e1b7df6, State: Initialized, Role: FOLLOWER
I20260925 02:35:22.328439 30075 consensus_queue.cc:260] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e764e9f5afd04036a8c6119c1e1b7df6" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35093 } }
I20260925 02:35:22.328518 30075 raft_consensus.cc:388] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:22.328547 30075 raft_consensus.cc:482] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:22.328588 30075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:22.329253 30075 raft_consensus.cc:504] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e764e9f5afd04036a8c6119c1e1b7df6" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35093 } }
I20260925 02:35:22.329345 30075 leader_election.cc:302] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e764e9f5afd04036a8c6119c1e1b7df6; no voters:
I20260925 02:35:22.329545 30075 leader_election.cc:288] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:22.329579 30080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:22.329727 30080 raft_consensus.cc:686] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [term 1 LEADER]: Becoming Leader. State: Replica: e764e9f5afd04036a8c6119c1e1b7df6, State: Running, Role: LEADER
I20260925 02:35:22.329847 30075 sys_catalog.cc:564] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:22.329879 30080 consensus_queue.cc:237] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e764e9f5afd04036a8c6119c1e1b7df6" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35093 } }
I20260925 02:35:22.330188 30080 sys_catalog.cc:455] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e764e9f5afd04036a8c6119c1e1b7df6. Latest consensus state: current_term: 1 leader_uuid: "e764e9f5afd04036a8c6119c1e1b7df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e764e9f5afd04036a8c6119c1e1b7df6" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35093 } } }
I20260925 02:35:22.330265 30080 sys_catalog.cc:458] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:22.330200 30081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e764e9f5afd04036a8c6119c1e1b7df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e764e9f5afd04036a8c6119c1e1b7df6" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 35093 } } }
I20260925 02:35:22.330417 30081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:22.332350 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.325497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.10.13.62:35093)
0925 02:35:22.325748 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.325757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.325791 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:35:22.326318 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.326327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.326408 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.326557 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:22.326751 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.328768 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.329826 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.330469 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.330593 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.331539 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:22.331544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:22.331545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:22.331751 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:22.331799 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:22.331815 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:22.331886 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.332084 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.332172 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
W20260925 02:35:22.331737 30096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:22.332713 30096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:22.332792 30085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:22.332981 30085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:22.334438 30085 catalog_manager.cc:1347] Generated new cluster ID: e4091bf673ac4c1db045fe6de1e9ae11
I20260925 02:35:22.334493 30085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:22.346899 30085 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:22.347517 30085 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:22.350497 30085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6: Generated new TSK 0
I20260925 02:35:22.350693 30085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:22.359088 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.1:0
--local_ip_for_outbound_sockets=127.10.13.1
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:35093
--builtin_ntp_servers=127.10.13.20:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:22.444945 30100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:22.445200 30100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:22.445238 30100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:22.446417 30100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:22.446487 30100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:22.446533 30100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.1
I20260925 02:35:22.447958 30100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.13.1
--webserver_port=0
--tserver_master_addrs=127.10.13.62:35093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.1
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:22.448246 30100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:22.448567 30100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:22.453790 30105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:22.453773 30106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:22.453790 30108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:22.454362 30100 server_base.cc:1034] running on GCE node
I20260925 02:35:22.454586 30100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:22.454852 30100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:22.456027 30100 hybrid_clock.cc:648] HybridClock initialized: now 1790303722456009 us; error 31 us; skew 500 ppm
I20260925 02:35:22.457502 30100 webserver.cc:460] Webserver started at http://127.10.13.1:34897/ using document root <none> and password file <none>
I20260925 02:35:22.457793 30100 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:22.457875 30100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:22.458042 30100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:22.459153 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/data/instance:
uuid: "202595da2744483b9a3b2e3abe219ab3"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "8f0b2c10005b7c5332bd007b282ab694"
server_key_iv: "edc85d3596eb7af162564346e22a287d"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.459544 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance:
uuid: "202595da2744483b9a3b2e3abe219ab3"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "8f0b2c10005b7c5332bd007b282ab694"
server_key_iv: "edc85d3596eb7af162564346e22a287d"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.461061 30100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:22.461912 30114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.462102 30100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:22.462215 30100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/wal
uuid: "202595da2744483b9a3b2e3abe219ab3"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "8f0b2c10005b7c5332bd007b282ab694"
server_key_iv: "edc85d3596eb7af162564346e22a287d"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.462292 30100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:22.486407 30100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:22.486780 30100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:22.487008 30100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:22.487288 30100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:22.487643 30100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:22.487699 30100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.487740 30100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:22.487769 30100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.494136 30100 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.1:37183
I20260925 02:35:22.494228 30228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.1:37183 every 8 connection(s)
I20260925 02:35:22.494642 30100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/data/info.pb
I20260925 02:35:22.499881 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.495459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.1:58017 (local address 127.10.13.62:35093)
0925 02:35:22.495551 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.495554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.495762 (+ 208us) server_negotiation.cc:407] Connection header received
0925 02:35:22.496261 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.496262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.496273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.496330 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:22.497022 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.497534 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.498489 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.498671 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.498722 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.499645 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:22.499646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:22.499646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:22.499651 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:22.499663 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:22.499666 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:22.499715 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.499785 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.499811 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:35:22.500133 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.495414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:35093 (local address 127.10.13.1:58017)
0925 02:35:22.495632 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.495669 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.496164 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.496392 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.496395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.496404 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:22.496937 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.496941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.497624 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.497625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.498404 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.498406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.498746 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.498747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.498860 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.499492 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:22.499493 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:22.499532 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:22.499559 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:22.499719 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:22.499722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:22.499724 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.499771 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.499860 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260925 02:35:22.500718 30229 heartbeater.cc:347] Connected to a master server at 127.10.13.62:35093
I20260925 02:35:22.500833 30229 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:22.501062 30229 heartbeater.cc:511] Master 127.10.13.62:35093 requested a full tablet report, sending...
I20260925 02:35:22.501663 30039 ts_manager.cc:194] Registered new tserver with Master: 202595da2744483b9a3b2e3abe219ab3 (127.10.13.1:37183)
I20260925 02:35:22.502293 30039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.1:58017
I20260925 02:35:22.504807 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 30100
I20260925 02:35:22.504974 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-0/wal/instance
I20260925 02:35:22.505206 10292 external_mini_cluster.cc:1428] Setting key a521063a2a71567918972a5102009cbe
I20260925 02:35:22.506142 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.2:0
--local_ip_for_outbound_sockets=127.10.13.2
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:35093
--builtin_ntp_servers=127.10.13.20:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:22.588961 30232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:22.589179 30232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:22.589201 30232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:22.590426 30232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:22.590478 30232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:22.590504 30232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.2
I20260925 02:35:22.591907 30232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.13.2
--webserver_port=0
--tserver_master_addrs=127.10.13.62:35093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.2
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:22.592171 30232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:22.592518 30232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:22.598132 30240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:22.598135 30238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:22.598162 30237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:22.598564 30232 server_base.cc:1034] running on GCE node
I20260925 02:35:22.598869 30232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:22.599140 30232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:22.600306 30232 hybrid_clock.cc:648] HybridClock initialized: now 1790303722600291 us; error 27 us; skew 500 ppm
I20260925 02:35:22.601686 30232 webserver.cc:460] Webserver started at http://127.10.13.2:33833/ using document root <none> and password file <none>
I20260925 02:35:22.601958 30232 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:22.602030 30232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:22.602185 30232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:22.603319 30232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/data/instance:
uuid: "2070df7af1a14909a69eb84abc6f2402"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "3b11b08f8f00680721c5aea39ff6d25e"
server_key_iv: "abeaa36ac3c7133f93a6895370008012"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.603761 30232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance:
uuid: "2070df7af1a14909a69eb84abc6f2402"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "3b11b08f8f00680721c5aea39ff6d25e"
server_key_iv: "abeaa36ac3c7133f93a6895370008012"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.605573 30232 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 02:35:22.606400 30246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.606596 30232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:22.606701 30232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/wal
uuid: "2070df7af1a14909a69eb84abc6f2402"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "3b11b08f8f00680721c5aea39ff6d25e"
server_key_iv: "abeaa36ac3c7133f93a6895370008012"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.606792 30232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:22.634557 30232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:22.634919 30232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:22.635094 30232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:22.635349 30232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:22.635697 30232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:22.635754 30232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.635790 30232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:22.635833 30232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.641752 30232 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.2:39225
I20260925 02:35:22.641798 30360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.2:39225 every 8 connection(s)
I20260925 02:35:22.642155 30232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/data/info.pb
I20260925 02:35:22.647620 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.642923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:50207 (local address 127.10.13.62:35093)
0925 02:35:22.642993 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.642995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.643273 (+ 278us) server_negotiation.cc:407] Connection header received
0925 02:35:22.643836 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.643837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.643851 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.643898 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:22.644604 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.645189 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.646210 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.646409 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.646489 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.647318 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:22.647320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:22.647320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:22.647325 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:22.647339 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:22.647344 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:22.647412 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.647512 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.647539 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:35:22.647922 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.642889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:35093 (local address 127.10.13.2:50207)
0925 02:35:22.643128 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.643167 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.643705 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.643970 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.643975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.643985 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:22.644503 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.644508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.645310 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.645311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.646111 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.646113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.646498 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.646499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.646642 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.647173 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:22.647174 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:22.647202 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:22.647224 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:22.647414 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:22.647420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:22.647423 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.647512 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.647616 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260925 02:35:22.648479 30361 heartbeater.cc:347] Connected to a master server at 127.10.13.62:35093
I20260925 02:35:22.648566 30361 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:22.648715 30361 heartbeater.cc:511] Master 127.10.13.62:35093 requested a full tablet report, sending...
I20260925 02:35:22.649132 30039 ts_manager.cc:194] Registered new tserver with Master: 2070df7af1a14909a69eb84abc6f2402 (127.10.13.2:39225)
I20260925 02:35:22.649521 30039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.2:50207
I20260925 02:35:22.651612 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 30232
I20260925 02:35:22.651703 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/wal/instance
I20260925 02:35:22.651866 10292 external_mini_cluster.cc:1428] Setting key 113b9aa5a52a422d0bef8489b5dcf874
I20260925 02:35:22.653020 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.13.3:0
--local_ip_for_outbound_sockets=127.10.13.3
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:35093
--builtin_ntp_servers=127.10.13.20:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:35:22.736384 30364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:22.736624 30364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:22.736661 30364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:22.737862 30364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:22.737932 30364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:22.737975 30364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.13.3
I20260925 02:35:22.739363 30364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.13.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.13.3
--webserver_port=0
--tserver_master_addrs=127.10.13.62:35093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.13.3
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:22.739638 30364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:22.739982 30364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:22.745240 30369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:22.745275 30364 server_base.cc:1034] running on GCE node
W20260925 02:35:22.745263 30370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:22.745271 30372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:22.745671 30364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:22.745925 30364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:22.747098 30364 hybrid_clock.cc:648] HybridClock initialized: now 1790303722747075 us; error 34 us; skew 500 ppm
I20260925 02:35:22.748382 30364 webserver.cc:460] Webserver started at http://127.10.13.3:35563/ using document root <none> and password file <none>
I20260925 02:35:22.748646 30364 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:22.748718 30364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:22.748908 30364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:22.750025 30364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/data/instance:
uuid: "2b860d954e66463e921bc2c999d96fd5"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "1992b3204cb239d2a6d230ba417824d9"
server_key_iv: "27c828aae262d1740ff6b96734233f3d"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.750393 30364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance:
uuid: "2b860d954e66463e921bc2c999d96fd5"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "1992b3204cb239d2a6d230ba417824d9"
server_key_iv: "27c828aae262d1740ff6b96734233f3d"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.751693 30364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:22.752465 30378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.752655 30364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:22.752760 30364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/wal
uuid: "2b860d954e66463e921bc2c999d96fd5"
format_stamp: "Formatted at 2026-09-25 02:35:22 on dist-test-slave-q8mg"
server_key: "1992b3204cb239d2a6d230ba417824d9"
server_key_iv: "27c828aae262d1740ff6b96734233f3d"
server_key_version: "encryptionkey@0"
I20260925 02:35:22.752844 30364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:22.761109 30364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:22.761400 30364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:22.761555 30364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:22.761780 30364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:35:22.762116 30364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:35:22.762174 30364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.762216 30364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:35:22.762244 30364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:22.768240 30364 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.3:34491
I20260925 02:35:22.768353 30492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.3:34491 every 8 connection(s)
I20260925 02:35:22.768666 30364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/data/info.pb
I20260925 02:35:22.774200 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.769515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.3:45187 (local address 127.10.13.62:35093)
0925 02:35:22.769590 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.769592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.769835 (+ 243us) server_negotiation.cc:407] Connection header received
0925 02:35:22.770334 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.770335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.770347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.770417 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:22.771124 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.771702 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.772811 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.773087 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.773159 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.773970 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:22.773972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:22.773972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:22.773976 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:22.773990 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:22.773994 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:22.774043 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.774115 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.774138 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:35:22.774376 30495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.769476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:35093 (local address 127.10.13.3:45187)
0925 02:35:22.769664 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.769706 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.770204 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.770495 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.770499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.770507 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:22.771024 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.771028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.771852 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.771853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.772669 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.772672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.773179 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.773181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.773297 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.773828 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:35:22.773829 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:22.773858 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:35:22.773880 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:22.774047 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:22.774049 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:22.774051 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.774099 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.774176 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260925 02:35:22.774971 30493 heartbeater.cc:347] Connected to a master server at 127.10.13.62:35093
I20260925 02:35:22.775081 30493 heartbeater.cc:464] Registering TS with master...
I20260925 02:35:22.775290 30493 heartbeater.cc:511] Master 127.10.13.62:35093 requested a full tablet report, sending...
I20260925 02:35:22.775708 30039 ts_manager.cc:194] Registered new tserver with Master: 2b860d954e66463e921bc2c999d96fd5 (127.10.13.3:34491)
I20260925 02:35:22.776081 30039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.13.3:45187
I20260925 02:35:22.778003 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 30364
I20260925 02:35:22.778124 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/wal/instance
I20260925 02:35:22.778337 10292 external_mini_cluster.cc:1428] Setting key 33b8990a669813f88cf81a906b520ef3
I20260925 02:35:22.779353 10292 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:35:22.783517 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.780763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.10.13.62:35093)
0925 02:35:22.780840 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.780843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.781019 (+ 176us) server_negotiation.cc:407] Connection header received
0925 02:35:22.781059 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.781060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.781069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.781113 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:22.781290 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.781757 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.782458 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.782633 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.782682 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.783291 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:22.783293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:22.783293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:22.783297 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:22.783309 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:22.783313 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:22.783358 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.783405 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.783424 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:35:22.785049 30039 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:35:22.786561 30039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:35:22.793506 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.789039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42352 (local address 127.10.13.2:39225)
0925 02:35:22.789237 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.789244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.789370 (+ 126us) server_negotiation.cc:407] Connection header received
0925 02:35:22.789430 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.789432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.789448 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.789559 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:22.789719 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.790875 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.792370 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.793076 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.793139 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.793213 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.793362 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.793403 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260925 02:35:22.793516 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.788785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37183 (local address 127.0.0.1:45688)
0925 02:35:22.789223 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.789231 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.789269 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.789981 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.789982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.789984 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:22.790017 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.790019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.791202 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.791204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.792126 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.792129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.793214 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.793216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.793281 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.793286 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.793369 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.793409 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260925 02:35:22.793530 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.788785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.2:39225 (local address 127.0.0.1:42352)
0925 02:35:22.789287 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.789296 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.789349 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.789565 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.789568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.789571 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:22.789638 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.789640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.790957 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.790958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.792283 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.792286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.793161 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.793162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.793242 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.793257 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.793350 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.793407 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":66,"thread_start_us":119,"threads_started":1}
I20260925 02:35:22.793805 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.789034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.10.13.1:37183)
0925 02:35:22.789449 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.789452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.789460 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:35:22.789492 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.789515 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.789531 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.789607 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:22.790082 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.791114 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.792212 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.792999 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.793074 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.793176 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.793639 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.793692 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":97,"threads_started":1}
I20260925 02:35:22.794353 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.788784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:34491 (local address 127.0.0.1:33972)
0925 02:35:22.789005 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.789048 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.789118 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.789923 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.789924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.789926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:22.789981 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.789983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.790902 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.790903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.792283 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.792286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.794100 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.794101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.794164 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.794168 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.794245 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.794287 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":51,"thread_start_us":97,"threads_started":1}
I20260925 02:35:22.794651 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.789069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.10.13.3:34491)
0925 02:35:22.789729 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.789733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.789741 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:35:22.789792 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.789794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.789809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.789915 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:22.790103 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.790795 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.792952 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.793904 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.793969 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.794065 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.794519 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.794568 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":80,"threads_started":1}
I20260925 02:35:22.794571 30294 tablet_service.cc:1467] Processing CreateTablet for tablet e7a03385abb646d893109ad713d83195 (DEFAULT_TABLE table=test-table [id=0836d9227a5f48848e262c7b01ec9009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:22.794574 30162 tablet_service.cc:1467] Processing CreateTablet for tablet e7a03385abb646d893109ad713d83195 (DEFAULT_TABLE table=test-table [id=0836d9227a5f48848e262c7b01ec9009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:22.794986 30162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a03385abb646d893109ad713d83195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:22.794986 30294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a03385abb646d893109ad713d83195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:22.795326 30426 tablet_service.cc:1467] Processing CreateTablet for tablet e7a03385abb646d893109ad713d83195 (DEFAULT_TABLE table=test-table [id=0836d9227a5f48848e262c7b01ec9009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:35:22.795640 30426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a03385abb646d893109ad713d83195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:22.797606 30507 tablet_bootstrap.cc:492] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Bootstrap starting.
I20260925 02:35:22.798220 30507 tablet_bootstrap.cc:654] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:22.798501 30508 tablet_bootstrap.cc:492] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3: Bootstrap starting.
I20260925 02:35:22.798540 30507 log.cc:824] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:22.799147 30510 tablet_bootstrap.cc:492] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: Bootstrap starting.
I20260925 02:35:22.799213 30508 tablet_bootstrap.cc:654] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:22.799495 30508 log.cc:824] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:22.799779 30510 tablet_bootstrap.cc:654] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:22.800084 30510 log.cc:824] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:22.800220 30508 tablet_bootstrap.cc:492] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3: No bootstrap required, opened a new log
I20260925 02:35:22.800276 30508 ts_tablet_manager.cc:1397] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:35:22.800905 30510 tablet_bootstrap.cc:492] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: No bootstrap required, opened a new log
I20260925 02:35:22.800997 30510 ts_tablet_manager.cc:1397] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:35:22.801409 30507 tablet_bootstrap.cc:492] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: No bootstrap required, opened a new log
I20260925 02:35:22.801476 30507 ts_tablet_manager.cc:1397] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:35:22.802531 30508 raft_consensus.cc:348] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.802651 30508 raft_consensus.cc:374] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:22.802671 30508 raft_consensus.cc:729] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202595da2744483b9a3b2e3abe219ab3, State: Initialized, Role: FOLLOWER
I20260925 02:35:22.802793 30508 consensus_queue.cc:260] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.803058 30508 ts_tablet_manager.cc:1428] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:22.803078 30229 heartbeater.cc:503] Master 127.10.13.62:35093 was elected leader, sending a full tablet report...
I20260925 02:35:22.803653 30510 raft_consensus.cc:348] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.803812 30510 raft_consensus.cc:374] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:22.803845 30510 raft_consensus.cc:729] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b860d954e66463e921bc2c999d96fd5, State: Initialized, Role: FOLLOWER
I20260925 02:35:22.803961 30510 consensus_queue.cc:260] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.804060 30507 raft_consensus.cc:348] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.804201 30507 raft_consensus.cc:374] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:22.804234 30507 raft_consensus.cc:729] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2070df7af1a14909a69eb84abc6f2402, State: Initialized, Role: FOLLOWER
I20260925 02:35:22.804241 30510 ts_tablet_manager.cc:1428] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:22.804272 30493 heartbeater.cc:503] Master 127.10.13.62:35093 was elected leader, sending a full tablet report...
I20260925 02:35:22.804339 30507 consensus_queue.cc:260] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.804592 30507 ts_tablet_manager.cc:1428] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:35:22.804615 30361 heartbeater.cc:503] Master 127.10.13.62:35093 was elected leader, sending a full tablet report...
I20260925 02:35:22.867901 30515 raft_consensus.cc:482] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:35:22.868064 30515 raft_consensus.cc:504] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.868489 30515 leader_election.cc:288] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 202595da2744483b9a3b2e3abe219ab3 (127.10.13.1:37183), 2b860d954e66463e921bc2c999d96fd5 (127.10.13.3:34491)
I20260925 02:35:22.871349 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.868706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:58765 (local address 127.10.13.1:37183)
0925 02:35:22.868753 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.868755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.868765 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:35:22.868857 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.868858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.868872 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.868962 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:22.869138 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.869660 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.870617 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.871006 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.871061 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.871070 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.871233 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.871279 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:35:22.871348 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.868631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37183 (local address 127.10.13.2:58765)
0925 02:35:22.868696 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.868710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.868761 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.868972 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.868973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.868975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:22.869059 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.869062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.869734 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.869735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.870524 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.870526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.871088 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.871089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.871161 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.871164 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.871221 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.871252 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:35:22.871443 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.868786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.13.2:45985 (local address 127.10.13.3:34491)
0925 02:35:22.868830 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.868832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.868953 (+ 121us) server_negotiation.cc:407] Connection header received
0925 02:35:22.869004 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.869006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.869015 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.869078 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:35:22.869254 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.869743 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.870681 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.871060 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.871132 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.871141 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.871320 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.871368 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:35:22.871495 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.868673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.3:34491 (local address 127.10.13.2:45985)
0925 02:35:22.868844 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.868855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:22.868944 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:22.869109 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:22.869110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:22.869112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:35:22.869187 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.869188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.869827 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.869828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.870592 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:22.870595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.871154 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.871155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:22.871240 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.871245 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:35:22.871300 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:35:22.871335 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260925 02:35:22.871829 30183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a03385abb646d893109ad713d83195" candidate_uuid: "2070df7af1a14909a69eb84abc6f2402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202595da2744483b9a3b2e3abe219ab3" is_pre_election: true
I20260925 02:35:22.871857 30446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a03385abb646d893109ad713d83195" candidate_uuid: "2070df7af1a14909a69eb84abc6f2402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b860d954e66463e921bc2c999d96fd5" is_pre_election: true
I20260925 02:35:22.871994 30446 raft_consensus.cc:2454] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2070df7af1a14909a69eb84abc6f2402 in term 0.
I20260925 02:35:22.871994 30183 raft_consensus.cc:2454] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2070df7af1a14909a69eb84abc6f2402 in term 0.
I20260925 02:35:22.872239 30249 leader_election.cc:302] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202595da2744483b9a3b2e3abe219ab3, 2070df7af1a14909a69eb84abc6f2402; no voters:
I20260925 02:35:22.872423 30515 raft_consensus.cc:2781] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:35:22.872538 30515 raft_consensus.cc:482] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:35:22.872576 30515 raft_consensus.cc:3037] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:22.873409 30515 raft_consensus.cc:504] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.873570 30515 leader_election.cc:288] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [CANDIDATE]: Term 1 election: Requested vote from peers 202595da2744483b9a3b2e3abe219ab3 (127.10.13.1:37183), 2b860d954e66463e921bc2c999d96fd5 (127.10.13.3:34491)
I20260925 02:35:22.873796 30183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a03385abb646d893109ad713d83195" candidate_uuid: "2070df7af1a14909a69eb84abc6f2402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202595da2744483b9a3b2e3abe219ab3"
I20260925 02:35:22.873808 30446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a03385abb646d893109ad713d83195" candidate_uuid: "2070df7af1a14909a69eb84abc6f2402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b860d954e66463e921bc2c999d96fd5"
I20260925 02:35:22.873895 30183 raft_consensus.cc:3037] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:22.873907 30446 raft_consensus.cc:3037] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:22.874804 30183 raft_consensus.cc:2454] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2070df7af1a14909a69eb84abc6f2402 in term 1.
I20260925 02:35:22.874804 30446 raft_consensus.cc:2454] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2070df7af1a14909a69eb84abc6f2402 in term 1.
I20260925 02:35:22.875018 30249 leader_election.cc:302] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202595da2744483b9a3b2e3abe219ab3, 2070df7af1a14909a69eb84abc6f2402; no voters:
I20260925 02:35:22.875129 30515 raft_consensus.cc:2781] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:22.875276 30515 raft_consensus.cc:686] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 1 LEADER]: Becoming Leader. State: Replica: 2070df7af1a14909a69eb84abc6f2402, State: Running, Role: LEADER
I20260925 02:35:22.875386 30515 consensus_queue.cc:237] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } }
I20260925 02:35:22.876304 30038 catalog_manager.cc:5495] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2070df7af1a14909a69eb84abc6f2402 (127.10.13.2). New cstate: current_term: 1 leader_uuid: "2070df7af1a14909a69eb84abc6f2402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2070df7af1a14909a69eb84abc6f2402" member_type: VOTER last_known_addr { host: "127.10.13.2" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 } health_report { overall_health: UNKNOWN } } }
I20260925 02:35:22.890008 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.887537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.10.13.2:39225)
0925 02:35:22.887588 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:22.887590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:22.887600 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:35:22.887661 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:22.887662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:22.887690 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:22.887767 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:22.887875 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.888343 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.889185 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:22.889380 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:22.889457 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:22.889617 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:22.889725 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:22.889814 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:22.889853 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 02:35:22.889872 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:35:22.891795 30446 raft_consensus.cc:1261] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 1 FOLLOWER]: Refusing update from remote peer 2070df7af1a14909a69eb84abc6f2402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:22.891798 30183 raft_consensus.cc:1261] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [term 1 FOLLOWER]: Refusing update from remote peer 2070df7af1a14909a69eb84abc6f2402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:22.892168 30517 consensus_queue.cc:1035] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b860d954e66463e921bc2c999d96fd5" member_type: VOTER last_known_addr { host: "127.10.13.3" port: 34491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:22.892302 30515 consensus_queue.cc:1035] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202595da2744483b9a3b2e3abe219ab3" member_type: VOTER last_known_addr { host: "127.10.13.1" port: 37183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:35:22.892714 30362 tablet.cc:2369] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:35:22.895467 30521 mvcc.cc:205] Tried to move back new op lower bound from 7333084048962179072 to 7333084048898379776. Current Snapshot: MvccSnapshot[applied={T|T < 7333084048962179072}]
I20260925 02:35:22.897284 30039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:22.897383 30039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:35:22.898168 30039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e764e9f5afd04036a8c6119c1e1b7df6: Sending DeleteTablet for 3 replicas of tablet e7a03385abb646d893109ad713d83195
I20260925 02:35:22.898491 30294 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a03385abb646d893109ad713d83195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:22 UTC) from {username='slave'} at 127.0.0.1:42352
I20260925 02:35:22.898507 30162 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a03385abb646d893109ad713d83195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:22 UTC) from {username='slave'} at 127.0.0.1:45688
I20260925 02:35:22.898519 30426 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a03385abb646d893109ad713d83195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:35:22 UTC) from {username='slave'} at 127.0.0.1:33972
I20260925 02:35:22.898720 30530 tablet_replica.cc:330] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: stopping tablet replica
I20260925 02:35:22.898809 30531 tablet_replica.cc:330] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3: stopping tablet replica
I20260925 02:35:22.898826 30530 raft_consensus.cc:2229] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:22.898885 30531 raft_consensus.cc:2229] T e7a03385abb646d893109ad713d83195 P 202595da2744483b9a3b2e3abe219ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:35:22.898924 30530 raft_consensus.cc:2258] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:22.899075 30529 tablet_replica.cc:330] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: stopping tablet replica
I20260925 02:35:22.899153 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 30100
I20260925 02:35:22.899191 30529 raft_consensus.cc:2229] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:35:22.899350 30529 raft_consensus.cc:2258] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:35:22.899396 30530 ts_tablet_manager.cc:1905] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:22.900372 30529 ts_tablet_manager.cc:1905] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:35:22.902347 30530 ts_tablet_manager.cc:1918] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:22.902376 30529 ts_tablet_manager.cc:1918] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:35:22.902449 30530 log.cc:1196] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-2/wal/wals/e7a03385abb646d893109ad713d83195
I20260925 02:35:22.902520 30529 log.cc:1196] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Deleting WAL directory at /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790303478591324-10292-0/minicluster-data/ts-1/wal/wals/e7a03385abb646d893109ad713d83195
I20260925 02:35:22.902720 30530 ts_tablet_manager.cc:1939] T e7a03385abb646d893109ad713d83195 P 2b860d954e66463e921bc2c999d96fd5: Deleting consensus metadata
I20260925 02:35:22.902881 30529 ts_tablet_manager.cc:1939] T e7a03385abb646d893109ad713d83195 P 2070df7af1a14909a69eb84abc6f2402: Deleting consensus metadata
I20260925 02:35:22.903190 30023 catalog_manager.cc:4841] TS 2b860d954e66463e921bc2c999d96fd5 (127.10.13.3:34491): tablet e7a03385abb646d893109ad713d83195 (table test-table [id=0836d9227a5f48848e262c7b01ec9009]) successfully deleted
I20260925 02:35:22.903273 30024 catalog_manager.cc:4841] TS 2070df7af1a14909a69eb84abc6f2402 (127.10.13.2:39225): tablet e7a03385abb646d893109ad713d83195 (table test-table [id=0836d9227a5f48848e262c7b01ec9009]) successfully deleted
W20260925 02:35:22.907325 30025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.13.1:37183 (error 108)
I20260925 02:35:22.907896 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:22.907744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.1:37183 (local address 127.0.0.1:45704)
0925 02:35:22.907811 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:22.907861 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.1:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:35:22.907966 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 30232
W20260925 02:35:22.909368 30025 catalog_manager.cc:4584] TS 202595da2744483b9a3b2e3abe219ab3 (127.10.13.1:37183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7a03385abb646d893109ad713d83195: Network error: Client connection negotiation failed: client connection to 127.10.13.1:37183: connect: Connection refused (error 111)
I20260925 02:35:22.913038 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 30364
I20260925 02:35:22.917699 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 30007
2026-09-25T02:35:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (745 ms)
[----------] 6 tests from EncryptionPolicyTest (8465 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:22 dist-test-slave-q8mg krb5kdc[30535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:22 dist-test-slave-q8mg krb5kdc[30535](info): set up 2 sockets
Sep 25 02:35:22 dist-test-slave-q8mg krb5kdc[30535](info): commencing operation
krb5kdc: starting...
W20260925 02:35:23.963392 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:24 dist-test-slave-q8mg krb5kdc[30535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:24.087553 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:24.171753 30548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:24.171978 30548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:24.172003 30548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:24.173230 30548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:24.173275 30548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:24.173290 30548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:24.173303 30548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:24.173316 30548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:24.174917 30548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:24.175194 30548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:24.175443 30548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:24.180529 30553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:24.180517 30556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:24.180517 30554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:24.180787 30548 server_base.cc:1034] running on GCE node
I20260925 02:35:24.181071 30548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:24.181370 30548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:24.182540 30548 hybrid_clock.cc:648] HybridClock initialized: now 1790303724182515 us; error 41 us; skew 500 ppm
Sep 25 02:35:24 dist-test-slave-q8mg krb5kdc[30535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:24.185132 30548 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:24.185657 30548 webserver.cc:460] Webserver started at http://127.0.0.1:38603/ using document root <none> and password file <none>
I20260925 02:35:24.185914 30548 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:24.185992 30548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:24.186146 30548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:24.187222 30548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "95c712c1688947b08516c324eefcd587"
format_stamp: "Formatted at 2026-09-25 02:35:24 on dist-test-slave-q8mg"
server_key: "bb51c3d481364625a475e27e16579bbf"
server_key_iv: "66f4364b6c8af3645f0f2d747c2677c0"
server_key_version: "encryptionkey@0"
I20260925 02:35:24.187584 30548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "95c712c1688947b08516c324eefcd587"
format_stamp: "Formatted at 2026-09-25 02:35:24 on dist-test-slave-q8mg"
server_key: "bb51c3d481364625a475e27e16579bbf"
server_key_iv: "66f4364b6c8af3645f0f2d747c2677c0"
server_key_version: "encryptionkey@0"
I20260925 02:35:24.188993 30548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:24.189741 30563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:24.189919 30548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:24.190037 30548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "95c712c1688947b08516c324eefcd587"
format_stamp: "Formatted at 2026-09-25 02:35:24 on dist-test-slave-q8mg"
server_key: "bb51c3d481364625a475e27e16579bbf"
server_key_iv: "66f4364b6c8af3645f0f2d747c2677c0"
server_key_version: "encryptionkey@0"
I20260925 02:35:24.190124 30548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:24.224681 30548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:24.226864 30548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:24.227042 30548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:24.231343 30548 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42877
I20260925 02:35:24.231418 30615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42877 every 8 connection(s)
I20260925 02:35:24.231791 30548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:24.232731 30616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:24.233165 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 30548
I20260925 02:35:24.233302 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:24.233515 10292 external_mini_cluster.cc:1428] Setting key 917be9feab1c6c0f8e5fc8543c7db195
I20260925 02:35:24.236343 30616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587: Bootstrap starting.
I20260925 02:35:24.237139 30616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:24.237464 30616 log.cc:824] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:24.238317 30616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587: No bootstrap required, opened a new log
I20260925 02:35:24.240993 30616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c712c1688947b08516c324eefcd587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
Sep 25 02:35:24 dist-test-slave-q8mg krb5kdc[30535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:35:24.241180 30616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:24.241293 30616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c712c1688947b08516c324eefcd587, State: Initialized, Role: FOLLOWER
I20260925 02:35:24.241474 30616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c712c1688947b08516c324eefcd587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260925 02:35:24.241588 30616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:24.241639 30616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:24.241693 30616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:24.242519 30616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c712c1688947b08516c324eefcd587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260925 02:35:24.242650 30616 leader_election.cc:302] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95c712c1688947b08516c324eefcd587; no voters:
I20260925 02:35:24.242856 30616 leader_election.cc:288] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:24.242933 30621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:24.243129 30621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [term 1 LEADER]: Becoming Leader. State: Replica: 95c712c1688947b08516c324eefcd587, State: Running, Role: LEADER
I20260925 02:35:24.243234 30616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:24.243279 30621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c712c1688947b08516c324eefcd587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260925 02:35:24.243649 30623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95c712c1688947b08516c324eefcd587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c712c1688947b08516c324eefcd587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } } }
I20260925 02:35:24.243747 30623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:24.243638 30621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c712c1688947b08516c324eefcd587. Latest consensus state: current_term: 1 leader_uuid: "95c712c1688947b08516c324eefcd587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c712c1688947b08516c324eefcd587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } } }
I20260925 02:35:24.243996 30621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:24.245033 30628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:24.245584 30628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:24.246459 30619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:24.234141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47707 (local address 127.0.0.1:42877)
0925 02:35:24.234375 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:24.234389 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:24.234430 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:35:24.235070 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:24.235081 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:24.235149 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:24.235308 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:24.235941 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:24.237471 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:24.238241 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:24.238913 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:24.239036 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:24.241785 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:24.241790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:24.241791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:24.241803 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:24.244009 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:24.244582 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:24.244586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:24.244586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:24.244696 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:24.245259 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:24.245260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:24.245260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:24.245587 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:24.245750 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:24.246196 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 02:35:24.246309 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260925 02:35:24.247373 30628 catalog_manager.cc:1347] Generated new cluster ID: 8263c5a501aa481b83e3d980e415c010
I20260925 02:35:24.247423 30628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:24.247642 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:24.257298 30619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:24.249232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57843 (local address 127.0.0.1:42877)
0925 02:35:24.249261 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:24.249263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:24.249602 (+ 339us) server_negotiation.cc:407] Connection header received
0925 02:35:24.249665 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:24.249666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:24.249676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:24.249728 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:24.250443 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:24.251165 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:24.252287 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:24.252611 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:24.252667 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:24.254306 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:24.254308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:24.254308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:24.254315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:24.255950 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:24.256416 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:24.256417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:24.256417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:24.256478 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:24.256728 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:24.256728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:24.256729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:24.256839 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:24.256954 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:24.257148 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 02:35:24.257210 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260925 02:35:24.262285 30628 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:24.263074 30628 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:24.284403 30628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95c712c1688947b08516c324eefcd587: Generated new TSK 0
I20260925 02:35:24.284672 30628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:24.295491 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 30548
W20260925 02:35:24.300640 30533 connection.cc:535] client connection to 127.0.0.1:42877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42877): Connection reset by peer (error 104)
2026-09-25T02:35:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:24 dist-test-slave-q8mg krb5kdc[30647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:24 dist-test-slave-q8mg krb5kdc[30647](info): set up 2 sockets
Sep 25 02:35:24 dist-test-slave-q8mg krb5kdc[30647](info): commencing operation
krb5kdc: starting...
W20260925 02:35:25.327409 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:25 dist-test-slave-q8mg krb5kdc[30647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:25.448382 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:25.531944 30660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:25.532152 30660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:25.532176 30660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:25.533463 30660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:25.533507 30660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:25.533522 30660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:25.533535 30660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:25.533583 30660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:25.535082 30660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:25.535349 30660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:25.535599 30660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:25.540772 30668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:25.540839 30666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:25.540772 30665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:25.540813 30660 server_base.cc:1034] running on GCE node
I20260925 02:35:25.541252 30660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:25.541571 30660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:25.542742 30660 hybrid_clock.cc:648] HybridClock initialized: now 1790303725542726 us; error 47 us; skew 500 ppm
Sep 25 02:35:25 dist-test-slave-q8mg krb5kdc[30647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:25.545518 30660 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:25.546041 30660 webserver.cc:460] Webserver started at http://127.0.0.1:45675/ using document root <none> and password file <none>
I20260925 02:35:25.546294 30660 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:25.546366 30660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:25.546520 30660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:25.547590 30660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4"
format_stamp: "Formatted at 2026-09-25 02:35:25 on dist-test-slave-q8mg"
server_key: "c8037b533fa823be41e8c4eeebd204f9"
server_key_iv: "c840ef97f3ddd410f4f2dbea21c05a8f"
server_key_version: "encryptionkey@0"
I20260925 02:35:25.547962 30660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4"
format_stamp: "Formatted at 2026-09-25 02:35:25 on dist-test-slave-q8mg"
server_key: "c8037b533fa823be41e8c4eeebd204f9"
server_key_iv: "c840ef97f3ddd410f4f2dbea21c05a8f"
server_key_version: "encryptionkey@0"
I20260925 02:35:25.549397 30660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:25.550136 30675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:25.550335 30660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:25.550449 30660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4"
format_stamp: "Formatted at 2026-09-25 02:35:25 on dist-test-slave-q8mg"
server_key: "c8037b533fa823be41e8c4eeebd204f9"
server_key_iv: "c840ef97f3ddd410f4f2dbea21c05a8f"
server_key_version: "encryptionkey@0"
I20260925 02:35:25.550526 30660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:25.562925 30660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:25.565428 30660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:25.565609 30660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:25.569893 30660 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32989
I20260925 02:35:25.569947 30728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32989 every 8 connection(s)
I20260925 02:35:25.570335 30660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:25.571218 30729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:25.573405 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 30660
I20260925 02:35:25.573560 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:25.573647 30729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4: Bootstrap starting.
I20260925 02:35:25.573810 10292 external_mini_cluster.cc:1428] Setting key e2295179158209946bc2eec4c1f82ed3
I20260925 02:35:25.574426 30729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:25.574767 30729 log.cc:824] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:25.575585 30729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4: No bootstrap required, opened a new log
I20260925 02:35:25.577803 30729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260925 02:35:25.577963 30729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:25.578012 30729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e30ea5f0c8a4fa398f4d8b9737285e4, State: Initialized, Role: FOLLOWER
I20260925 02:35:25.578164 30729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260925 02:35:25.578246 30729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:25.578290 30729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:25.578344 30729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:25.579054 30729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260925 02:35:25.579185 30729 leader_election.cc:302] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e30ea5f0c8a4fa398f4d8b9737285e4; no voters:
I20260925 02:35:25.579382 30729 leader_election.cc:288] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:25.579403 30734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:25.579524 30734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [term 1 LEADER]: Becoming Leader. State: Replica: 9e30ea5f0c8a4fa398f4d8b9737285e4, State: Running, Role: LEADER
I20260925 02:35:25.579684 30729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:25.579661 30734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260925 02:35:25.580073 30732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:25.574366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38377 (local address 127.0.0.1:32989)
0925 02:35:25.574556 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:25.574564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:25.574591 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:35:25.575150 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:25.575160 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:25.575227 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:25.575360 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:25.575594 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:25.576504 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:25.577706 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:25.578235 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:25.578359 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:25.579184 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:25.579189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:25.579189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:25.579407 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:25.579472 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:25.579489 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:25.579579 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:25.579812 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:35:25.579839 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260925 02:35:25.580175 30735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e30ea5f0c8a4fa398f4d8b9737285e4. Latest consensus state: current_term: 1 leader_uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } } }
I20260925 02:35:25.580258 30735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:25.580423 30736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e30ea5f0c8a4fa398f4d8b9737285e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } } }
I20260925 02:35:25.580487 30736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:25.580780 30738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:25.581243 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:25.581313 30738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:25.583304 30738 catalog_manager.cc:1347] Generated new cluster ID: 6d020fa257b44d678db43efc955a113b
I20260925 02:35:25.583359 30738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:25.585850 30732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:25.582889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42999 (local address 127.0.0.1:32989)
0925 02:35:25.582944 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:25.582947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:25.582982 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:35:25.583060 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:25.583061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:25.583071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:25.583144 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:25.583291 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:25.583832 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:25.584572 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:25.584786 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:25.584906 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:25.585577 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:25.585578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:25.585579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:25.585583 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:25.585598 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:25.585602 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:25.585675 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:25.585736 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 02:35:25.585746 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:35:25.601243 30738 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:25.601886 30738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:25.611490 30738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e30ea5f0c8a4fa398f4d8b9737285e4: Generated new TSK 0
I20260925 02:35:25.611721 30738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:25.622668 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 30660
W20260925 02:35:25.627769 30645 connection.cc:535] client connection to 127.0.0.1:32989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32989): Connection reset by peer (error 104)
2026-09-25T02:35:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:25 dist-test-slave-q8mg krb5kdc[30761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:25 dist-test-slave-q8mg krb5kdc[30761](info): set up 2 sockets
Sep 25 02:35:25 dist-test-slave-q8mg krb5kdc[30761](info): commencing operation
krb5kdc: starting...
W20260925 02:35:26.667484 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:26 dist-test-slave-q8mg krb5kdc[30761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:26.791831 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:26.873577 30774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:26.873806 30774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:26.873831 30774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:26.874938 30774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:26.874976 30774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:26.874991 30774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:26.875003 30774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:26.875015 30774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:26.876447 30774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:26.876650 30774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:26.876823 30774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:35:26.881789 30774 server_base.cc:1034] running on GCE node
W20260925 02:35:26.881783 30779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:26.881758 30780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:26.881748 30782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:26.882193 30774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:26.882548 30774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:26.883728 30774 hybrid_clock.cc:648] HybridClock initialized: now 1790303726883719 us; error 36 us; skew 500 ppm
Sep 25 02:35:26 dist-test-slave-q8mg krb5kdc[30761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:26.893837 30774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:26.894381 30774 webserver.cc:460] Webserver started at http://127.0.0.1:37439/ using document root <none> and password file <none>
I20260925 02:35:26.894622 30774 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:26.894693 30774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:26.894855 30774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:26.895951 30774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "59d9bddc9fe14cb497d44f2723d8c94a"
format_stamp: "Formatted at 2026-09-25 02:35:26 on dist-test-slave-q8mg"
server_key: "b6b511042994289876765fb05edabd7f"
server_key_iv: "a13a2b7edf193108bb8719e2edc94167"
server_key_version: "encryptionkey@0"
I20260925 02:35:26.896320 30774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "59d9bddc9fe14cb497d44f2723d8c94a"
format_stamp: "Formatted at 2026-09-25 02:35:26 on dist-test-slave-q8mg"
server_key: "b6b511042994289876765fb05edabd7f"
server_key_iv: "a13a2b7edf193108bb8719e2edc94167"
server_key_version: "encryptionkey@0"
I20260925 02:35:26.897933 30774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:26.898759 30789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:26.898977 30774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:26.899086 30774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "59d9bddc9fe14cb497d44f2723d8c94a"
format_stamp: "Formatted at 2026-09-25 02:35:26 on dist-test-slave-q8mg"
server_key: "b6b511042994289876765fb05edabd7f"
server_key_iv: "a13a2b7edf193108bb8719e2edc94167"
server_key_version: "encryptionkey@0"
I20260925 02:35:26.899179 30774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:26.902685 30774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:26.905071 30774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:26.905230 30774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:26.909237 30774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41077
I20260925 02:35:26.909268 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41077 every 8 connection(s)
I20260925 02:35:26.909607 30774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:26.910605 30843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:26.912858 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a: Bootstrap starting.
I20260925 02:35:26.913499 30843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:26.913797 30843 log.cc:824] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:26.914474 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a: No bootstrap required, opened a new log
I20260925 02:35:26.916014 30843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d9bddc9fe14cb497d44f2723d8c94a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } }
I20260925 02:35:26.916148 30843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:26.916196 30843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d9bddc9fe14cb497d44f2723d8c94a, State: Initialized, Role: FOLLOWER
I20260925 02:35:26.916366 30843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d9bddc9fe14cb497d44f2723d8c94a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } }
I20260925 02:35:26.916431 30843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:26.916471 30843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:26.916520 30843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:26.916726 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 30774
I20260925 02:35:26.916837 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:26.917043 10292 external_mini_cluster.cc:1428] Setting key 9c9f3b2e03be02b25c5c759a74f09755
I20260925 02:35:26.917256 30843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d9bddc9fe14cb497d44f2723d8c94a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } }
I20260925 02:35:26.917383 30843 leader_election.cc:302] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59d9bddc9fe14cb497d44f2723d8c94a; no voters:
I20260925 02:35:26.917565 30843 leader_election.cc:288] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:26.917619 30847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:26.917896 30843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:26.918041 30847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [term 1 LEADER]: Becoming Leader. State: Replica: 59d9bddc9fe14cb497d44f2723d8c94a, State: Running, Role: LEADER
I20260925 02:35:26.918133 30847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d9bddc9fe14cb497d44f2723d8c94a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } }
I20260925 02:35:26.918862 30847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59d9bddc9fe14cb497d44f2723d8c94a. Latest consensus state: current_term: 1 leader_uuid: "59d9bddc9fe14cb497d44f2723d8c94a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d9bddc9fe14cb497d44f2723d8c94a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } } }
I20260925 02:35:26.918946 30847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:26.919075 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59d9bddc9fe14cb497d44f2723d8c94a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d9bddc9fe14cb497d44f2723d8c94a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } } }
I20260925 02:35:26.919148 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:26.919258 30857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:26.919584 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:26.917716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49735 (local address 127.0.0.1:41077)
0925 02:35:26.917891 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:26.917901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:26.917933 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:35:26.918057 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:26.918064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:26.918096 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:26.918511 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:26.918995 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:26.918998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:26.918999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:26.919003 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:26.919049 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:26.919065 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:26.919132 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:26.919345 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:35:26.919390 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260925 02:35:26.919811 30857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:26.920564 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:26.921782 30857 catalog_manager.cc:1347] Generated new cluster ID: 314158c9e2f941ae9951b8245897a146
I20260925 02:35:26.921846 30857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:26.926575 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:26.925415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53943 (local address 127.0.0.1:41077)
0925 02:35:26.925451 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:26.925453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:26.925570 (+ 117us) server_negotiation.cc:407] Connection header received
0925 02:35:26.925613 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:26.925614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:26.925620 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:26.925701 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:26.926322 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:26.926323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:26.926324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:26.926328 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:26.926347 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:26.926359 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:26.926425 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:26.926480 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 02:35:26.926494 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:35:26.928279 30857 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:26.928444 30857 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:26.941886 30857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59d9bddc9fe14cb497d44f2723d8c94a: Generated new TSK 0
I20260925 02:35:26.942104 30857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:26.947557 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 30774
2026-09-25T02:35:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:27 dist-test-slave-q8mg krb5kdc[30874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:27 dist-test-slave-q8mg krb5kdc[30874](info): set up 2 sockets
Sep 25 02:35:27 dist-test-slave-q8mg krb5kdc[30874](info): commencing operation
krb5kdc: starting...
W20260925 02:35:28.027384 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:28 dist-test-slave-q8mg krb5kdc[30874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:28.148229 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:28.230273 30887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:28.230520 30887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:28.230558 30887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:28.232300 30887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:28.232366 30887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:28.232393 30887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:28.232424 30887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:28.232452 30887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:28.233973 30887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:28.234253 30887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:28.234504 30887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:28.239419 30893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:28.239414 30892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:28.239416 30895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:28.239562 30887 server_base.cc:1034] running on GCE node
I20260925 02:35:28.239915 30887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:28.240164 30887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:28.241329 30887 hybrid_clock.cc:648] HybridClock initialized: now 1790303728241315 us; error 46 us; skew 500 ppm
Sep 25 02:35:28 dist-test-slave-q8mg krb5kdc[30874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:28.243839 30887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:28.244308 30887 webserver.cc:460] Webserver started at http://127.0.0.1:38373/ using document root <none> and password file <none>
I20260925 02:35:28.244499 30887 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:28.244544 30887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:28.244632 30887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:28.245743 30887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "059ad742331b4a448b051db61077dcf3"
format_stamp: "Formatted at 2026-09-25 02:35:28 on dist-test-slave-q8mg"
server_key: "15261bfadf4a13d22b3ecbb38d6397a4"
server_key_iv: "32625d3d4db20860c659f019eacb7c25"
server_key_version: "encryptionkey@0"
I20260925 02:35:28.246054 30887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "059ad742331b4a448b051db61077dcf3"
format_stamp: "Formatted at 2026-09-25 02:35:28 on dist-test-slave-q8mg"
server_key: "15261bfadf4a13d22b3ecbb38d6397a4"
server_key_iv: "32625d3d4db20860c659f019eacb7c25"
server_key_version: "encryptionkey@0"
I20260925 02:35:28.247332 30887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:35:28.248045 30903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:28.248227 30887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:35:28.248315 30887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "059ad742331b4a448b051db61077dcf3"
format_stamp: "Formatted at 2026-09-25 02:35:28 on dist-test-slave-q8mg"
server_key: "15261bfadf4a13d22b3ecbb38d6397a4"
server_key_iv: "32625d3d4db20860c659f019eacb7c25"
server_key_version: "encryptionkey@0"
I20260925 02:35:28.248374 30887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:28.267011 30887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:28.269183 30887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:28.269300 30887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:28.273442 30887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43259
I20260925 02:35:28.273505 30955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43259 every 8 connection(s)
I20260925 02:35:28.273895 30887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:28.274808 30956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:28.277289 30956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3: Bootstrap starting.
I20260925 02:35:28.277920 30956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:28.278228 30956 log.cc:824] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:28.279014 30956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3: No bootstrap required, opened a new log
I20260925 02:35:28.280627 30956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ad742331b4a448b051db61077dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260925 02:35:28.280790 30956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:28.280850 30956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059ad742331b4a448b051db61077dcf3, State: Initialized, Role: FOLLOWER
I20260925 02:35:28.281059 30956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ad742331b4a448b051db61077dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260925 02:35:28.281147 30956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:28.281188 30956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:28.281243 30956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:28.281771 30956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ad742331b4a448b051db61077dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260925 02:35:28.281894 30956 leader_election.cc:302] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 059ad742331b4a448b051db61077dcf3; no voters:
I20260925 02:35:28.282089 30956 leader_election.cc:288] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:28.282138 30959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:28.282357 30959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [term 1 LEADER]: Becoming Leader. State: Replica: 059ad742331b4a448b051db61077dcf3, State: Running, Role: LEADER
I20260925 02:35:28.282361 30956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:28.282461 30959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ad742331b4a448b051db61077dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260925 02:35:28.282889 30961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059ad742331b4a448b051db61077dcf3. Latest consensus state: current_term: 1 leader_uuid: "059ad742331b4a448b051db61077dcf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ad742331b4a448b051db61077dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } } }
I20260925 02:35:28.282977 30961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:28.282876 30960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "059ad742331b4a448b051db61077dcf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ad742331b4a448b051db61077dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } } }
I20260925 02:35:28.283116 30960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:28.283377 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 30887
I20260925 02:35:28.283504 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:28.283519 30963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:28.283708 10292 external_mini_cluster.cc:1428] Setting key 3f0c31d0f56039f80114e199a749bd8e
I20260925 02:35:28.284338 30963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:28.286132 30963 catalog_manager.cc:1347] Generated new cluster ID: 0c99261188234bcfbaba55acabea58c2
I20260925 02:35:28.286228 30963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:35:28 dist-test-slave-q8mg krb5kdc[30874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:35:28.292558 30963 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:28.293285 30963 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:28.294821 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:28.284417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51617 (local address 127.0.0.1:43259)
0925 02:35:28.284598 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:28.284609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:28.284643 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:35:28.284781 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:28.284787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:28.284832 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:28.285009 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:28.285698 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:28.286756 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:28.287546 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:28.287977 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:28.288098 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:28.290542 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:28.290546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:28.290546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:28.290562 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:28.293230 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:28.293607 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:28.293611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:28.293611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:28.293723 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:28.293940 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:28.293941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:28.293941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:28.294171 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:28.294342 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:28.294444 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:35:28.294604 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260925 02:35:28.295792 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:28.298954 30963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 059ad742331b4a448b051db61077dcf3: Generated new TSK 0
I20260925 02:35:28.299132 30963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:28.303503 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:28.297145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55383 (local address 127.0.0.1:43259)
0925 02:35:28.297188 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:28.297190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:28.297288 (+ 98us) server_negotiation.cc:407] Connection header received
0925 02:35:28.297359 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:28.297361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:28.297371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:28.297419 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:28.297929 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:28.298444 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:28.299446 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:28.299675 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:28.299767 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:28.300991 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:28.300992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:28.300993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:28.301001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:28.302261 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:28.302608 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:28.302610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:28.302610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:28.302670 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:28.302991 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:28.302992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:28.302993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:28.303108 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:28.303210 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:28.303319 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:35:28.303401 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:35:28.305449 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 30887
2026-09-25T02:35:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:28 dist-test-slave-q8mg krb5kdc[30987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:28 dist-test-slave-q8mg krb5kdc[30987](info): set up 2 sockets
Sep 25 02:35:28 dist-test-slave-q8mg krb5kdc[30987](info): commencing operation
krb5kdc: starting...
W20260925 02:35:29.347334 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:29 dist-test-slave-q8mg krb5kdc[30987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:29.471733 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:29.552438 31000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:29.552687 31000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:29.552726 31000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:29.553951 31000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:29.554026 31000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:29.554050 31000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:29.554072 31000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:29.554101 31000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:29.555609 31000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:29.555903 31000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:29.556145 31000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:29.561149 31006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:29.561160 31005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:29.561180 31000 server_base.cc:1034] running on GCE node
W20260925 02:35:29.561158 31008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:29.561602 31000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:29.561919 31000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:29.563086 31000 hybrid_clock.cc:648] HybridClock initialized: now 1790303729563074 us; error 48 us; skew 500 ppm
Sep 25 02:35:29 dist-test-slave-q8mg krb5kdc[30987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:29.565569 31000 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:29.566051 31000 webserver.cc:460] Webserver started at http://127.0.0.1:41259/ using document root <none> and password file <none>
I20260925 02:35:29.566295 31000 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:29.566365 31000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:29.566516 31000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:29.567595 31000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "346bc2d5c6c14577bfe7e499000632ae"
format_stamp: "Formatted at 2026-09-25 02:35:29 on dist-test-slave-q8mg"
server_key: "588d390479cd71e9f4c3bb3b9afd2cf6"
server_key_iv: "04636035383cb2201c62d7049b6db4b7"
server_key_version: "encryptionkey@0"
I20260925 02:35:29.567953 31000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "346bc2d5c6c14577bfe7e499000632ae"
format_stamp: "Formatted at 2026-09-25 02:35:29 on dist-test-slave-q8mg"
server_key: "588d390479cd71e9f4c3bb3b9afd2cf6"
server_key_iv: "04636035383cb2201c62d7049b6db4b7"
server_key_version: "encryptionkey@0"
I20260925 02:35:29.569377 31000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:29.570133 31015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:29.570322 31000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:35:29.570434 31000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "346bc2d5c6c14577bfe7e499000632ae"
format_stamp: "Formatted at 2026-09-25 02:35:29 on dist-test-slave-q8mg"
server_key: "588d390479cd71e9f4c3bb3b9afd2cf6"
server_key_iv: "04636035383cb2201c62d7049b6db4b7"
server_key_version: "encryptionkey@0"
I20260925 02:35:29.570509 31000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:29.586200 31000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:29.588590 31000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:29.588761 31000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:29.592816 31000 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46843
I20260925 02:35:29.592902 31068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46843 every 8 connection(s)
I20260925 02:35:29.593257 31000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:29.594146 31069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:29.596504 31069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae: Bootstrap starting.
I20260925 02:35:29.596519 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31000
I20260925 02:35:29.596652 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:29.596833 10292 external_mini_cluster.cc:1428] Setting key 72a7132e53e75bc3dee99111b0d706dc
I20260925 02:35:29.597287 31069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:29.597679 31069 log.cc:824] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:29.598457 31069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae: No bootstrap required, opened a new log
I20260925 02:35:29.600679 31069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346bc2d5c6c14577bfe7e499000632ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260925 02:35:29.600816 31069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:29.600838 31069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346bc2d5c6c14577bfe7e499000632ae, State: Initialized, Role: FOLLOWER
I20260925 02:35:29.600997 31069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346bc2d5c6c14577bfe7e499000632ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260925 02:35:29.601073 31069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:29.601095 31069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:29.601164 31069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:29.601857 31069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346bc2d5c6c14577bfe7e499000632ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260925 02:35:29.602005 31069 leader_election.cc:302] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 346bc2d5c6c14577bfe7e499000632ae; no voters:
I20260925 02:35:29.602224 31069 leader_election.cc:288] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:29.602246 31074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:29.602362 31074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [term 1 LEADER]: Becoming Leader. State: Replica: 346bc2d5c6c14577bfe7e499000632ae, State: Running, Role: LEADER
I20260925 02:35:29.602451 31074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346bc2d5c6c14577bfe7e499000632ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260925 02:35:29.602515 31069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:29.602767 31075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "346bc2d5c6c14577bfe7e499000632ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346bc2d5c6c14577bfe7e499000632ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } } }
I20260925 02:35:29.602872 31075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:29.602955 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:29.597379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42939 (local address 127.0.0.1:46843)
0925 02:35:29.597568 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:29.597576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:29.597601 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:35:29.598076 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:29.598084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:29.598133 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:29.598247 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:29.598469 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:29.599391 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:29.600592 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:29.601155 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:29.601268 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:29.601986 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:29.601991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:29.601991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:29.602220 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:29.602253 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:29.602280 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:29.602604 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:29.602641 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 02:35:29.602704 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260925 02:35:29.603318 31079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:29.603847 31079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:29.604202 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:29.604228 31076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 346bc2d5c6c14577bfe7e499000632ae. Latest consensus state: current_term: 1 leader_uuid: "346bc2d5c6c14577bfe7e499000632ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346bc2d5c6c14577bfe7e499000632ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } } }
I20260925 02:35:29.604341 31076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:29.605611 31079 catalog_manager.cc:1347] Generated new cluster ID: 136e354347404ea8ac02c94cd270a1e9
I20260925 02:35:29.605665 31079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:29.608455 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:29.605614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60221 (local address 127.0.0.1:46843)
0925 02:35:29.605647 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:29.605649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:29.605858 (+ 209us) server_negotiation.cc:407] Connection header received
0925 02:35:29.605918 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:29.605919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:29.605930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:29.605979 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:29.606096 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:29.606568 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:29.607259 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:29.607486 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:29.607562 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:29.608217 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:29.608218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:29.608218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:29.608223 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:29.608236 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:29.608240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:29.608286 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:29.608341 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 02:35:29.608367 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:35:29.631095 31079 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:29.631964 31079 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:29.635360 31079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 346bc2d5c6c14577bfe7e499000632ae: Generated new TSK 0
I20260925 02:35:29.635548 31079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:29.644717 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31000
2026-09-25T02:35:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:29 dist-test-slave-q8mg krb5kdc[31100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:29 dist-test-slave-q8mg krb5kdc[31100](info): set up 2 sockets
Sep 25 02:35:29 dist-test-slave-q8mg krb5kdc[31100](info): commencing operation
krb5kdc: starting...
W20260925 02:35:30.695523 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:30 dist-test-slave-q8mg krb5kdc[31100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:30.818065 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:30.901403 31113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:30.901626 31113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:30.901651 31113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:30.902937 31113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:30.902984 31113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:30.902998 31113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:30.903014 31113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:30.903029 31113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:30.904577 31113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:30.904861 31113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:30.905174 31113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:30.910404 31121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:30.910414 31119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:30.910404 31118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:30.910547 31113 server_base.cc:1034] running on GCE node
I20260925 02:35:30.910907 31113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:30.911216 31113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:30.912382 31113 hybrid_clock.cc:648] HybridClock initialized: now 1790303730912359 us; error 40 us; skew 500 ppm
Sep 25 02:35:30 dist-test-slave-q8mg krb5kdc[31100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:30.915143 31113 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:30.915592 31113 webserver.cc:460] Webserver started at http://127.0.0.1:45767/ using document root <none> and password file <none>
I20260925 02:35:30.915833 31113 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:30.915901 31113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:30.916052 31113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:30.917255 31113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "813feb5e36eb4d54b5b4d35ed9509e2f"
format_stamp: "Formatted at 2026-09-25 02:35:30 on dist-test-slave-q8mg"
server_key: "340e4977c2a6e51e9ec722e39c8c5eec"
server_key_iv: "51d2f57be51913f69fbf4a53607ed660"
server_key_version: "encryptionkey@0"
I20260925 02:35:30.917622 31113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "813feb5e36eb4d54b5b4d35ed9509e2f"
format_stamp: "Formatted at 2026-09-25 02:35:30 on dist-test-slave-q8mg"
server_key: "340e4977c2a6e51e9ec722e39c8c5eec"
server_key_iv: "51d2f57be51913f69fbf4a53607ed660"
server_key_version: "encryptionkey@0"
I20260925 02:35:30.919057 31113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:35:30.919936 31128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:30.920137 31113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:30.920266 31113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "813feb5e36eb4d54b5b4d35ed9509e2f"
format_stamp: "Formatted at 2026-09-25 02:35:30 on dist-test-slave-q8mg"
server_key: "340e4977c2a6e51e9ec722e39c8c5eec"
server_key_iv: "51d2f57be51913f69fbf4a53607ed660"
server_key_version: "encryptionkey@0"
I20260925 02:35:30.920344 31113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:30.923806 31113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:30.926151 31113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:30.926316 31113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:30.930334 31113 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38679
I20260925 02:35:30.930387 31180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38679 every 8 connection(s)
I20260925 02:35:30.930760 31113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:30.931828 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:30.933305 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31113
I20260925 02:35:30.933439 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:30.933641 10292 external_mini_cluster.cc:1428] Setting key 1e24635de88ccf34b4ed08c9b6a674c6
I20260925 02:35:30.934859 31181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f: Bootstrap starting.
I20260925 02:35:30.935670 31181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:30.936043 31181 log.cc:824] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:30.936440 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:30.934265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53673 (local address 127.0.0.1:38679)
0925 02:35:30.934465 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:30.934473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:30.934499 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:35:30.934896 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:30.934902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:30.934969 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:30.935102 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:30.935851 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:30.935856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:30.935856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:30.935860 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:30.935914 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:30.935925 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:30.935997 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:30.936253 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:35:30.936295 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260925 02:35:30.936836 31181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f: No bootstrap required, opened a new log
I20260925 02:35:30.937685 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:30.939613 31181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813feb5e36eb4d54b5b4d35ed9509e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260925 02:35:30.939813 31181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:30.939893 31181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813feb5e36eb4d54b5b4d35ed9509e2f, State: Initialized, Role: FOLLOWER
I20260925 02:35:30.940055 31181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813feb5e36eb4d54b5b4d35ed9509e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260925 02:35:30.940165 31181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:30.940212 31181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:30.940264 31181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:30.940506 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:30.939391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38723 (local address 127.0.0.1:38679)
0925 02:35:30.939426 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:30.939434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:30.939530 (+ 96us) server_negotiation.cc:407] Connection header received
0925 02:35:30.939570 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:30.939571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:30.939575 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:30.939641 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:30.940285 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:30.940286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:30.940287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:30.940291 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:30.940306 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:30.940311 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:30.940380 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:30.940439 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:35:30.940454 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:35:30.940979 31181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813feb5e36eb4d54b5b4d35ed9509e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260925 02:35:30.941104 31181 leader_election.cc:302] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 813feb5e36eb4d54b5b4d35ed9509e2f; no voters:
I20260925 02:35:30.941265 31181 leader_election.cc:288] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:30.941332 31192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:30.941488 31192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [term 1 LEADER]: Becoming Leader. State: Replica: 813feb5e36eb4d54b5b4d35ed9509e2f, State: Running, Role: LEADER
I20260925 02:35:30.941529 31181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:30.941639 31192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813feb5e36eb4d54b5b4d35ed9509e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260925 02:35:30.942045 31194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 813feb5e36eb4d54b5b4d35ed9509e2f. Latest consensus state: current_term: 1 leader_uuid: "813feb5e36eb4d54b5b4d35ed9509e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813feb5e36eb4d54b5b4d35ed9509e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } } }
I20260925 02:35:30.942148 31194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:30.942719 31193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "813feb5e36eb4d54b5b4d35ed9509e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813feb5e36eb4d54b5b4d35ed9509e2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } } }
I20260925 02:35:30.942795 31193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f [sys.catalog]: This master's current role is: LEADER
W20260925 02:35:30.943341 31208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:30.943387 31208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:30.943454 31201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:30.943612 31201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:30.945047 31201 catalog_manager.cc:1347] Generated new cluster ID: e47c2245c0d24fef943bba8c51daa953
I20260925 02:35:30.945099 31201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:30.951126 31201 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:30.951323 31201 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:30.955106 31201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 813feb5e36eb4d54b5b4d35ed9509e2f: Generated new TSK 0
I20260925 02:35:30.955305 31201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:30.958717 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31113
2026-09-25T02:35:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:31 dist-test-slave-q8mg krb5kdc[31213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:31 dist-test-slave-q8mg krb5kdc[31213](info): set up 2 sockets
Sep 25 02:35:31 dist-test-slave-q8mg krb5kdc[31213](info): commencing operation
krb5kdc: starting...
W20260925 02:35:32.007863 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:32 dist-test-slave-q8mg krb5kdc[31213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:32.131383 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:32.214911 31226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:32.215142 31226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:32.215167 31226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:32.216435 31226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:32.216485 31226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:32.216498 31226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:32.216511 31226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:32.216523 31226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:32.218107 31226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:32.218391 31226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:32.218641 31226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:32.223662 31232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:32.223664 31234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:32.223639 31231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:32.223941 31226 server_base.cc:1034] running on GCE node
I20260925 02:35:32.224162 31226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:32.224422 31226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:32.225565 31226 hybrid_clock.cc:648] HybridClock initialized: now 1790303732225551 us; error 29 us; skew 500 ppm
Sep 25 02:35:32 dist-test-slave-q8mg krb5kdc[31213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:32.233965 31226 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:32.234380 31226 webserver.cc:460] Webserver started at http://127.0.0.1:44589/ using document root <none> and password file <none>
I20260925 02:35:32.234573 31226 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:32.234615 31226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:32.234706 31226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:32.235742 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "2c896ef7fde64b348c4987be488d98da"
format_stamp: "Formatted at 2026-09-25 02:35:32 on dist-test-slave-q8mg"
server_key: "e0e3d7b97e7fb87708cdd5bd74a67fab"
server_key_iv: "b2399ab6dddf256b69b25346205b2a0c"
server_key_version: "encryptionkey@0"
I20260925 02:35:32.236055 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "2c896ef7fde64b348c4987be488d98da"
format_stamp: "Formatted at 2026-09-25 02:35:32 on dist-test-slave-q8mg"
server_key: "e0e3d7b97e7fb87708cdd5bd74a67fab"
server_key_iv: "b2399ab6dddf256b69b25346205b2a0c"
server_key_version: "encryptionkey@0"
I20260925 02:35:32.237524 31226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 02:35:32.238374 31241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:32.238584 31226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:32.238714 31226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "2c896ef7fde64b348c4987be488d98da"
format_stamp: "Formatted at 2026-09-25 02:35:32 on dist-test-slave-q8mg"
server_key: "e0e3d7b97e7fb87708cdd5bd74a67fab"
server_key_iv: "b2399ab6dddf256b69b25346205b2a0c"
server_key_version: "encryptionkey@0"
I20260925 02:35:32.238796 31226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:32.267964 31226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:32.270370 31226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:32.270551 31226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:32.274847 31226 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36245
I20260925 02:35:32.274909 31293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36245 every 8 connection(s)
I20260925 02:35:32.275290 31226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:32.276175 31294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:32.276911 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31226
I20260925 02:35:32.277072 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:32.277262 10292 external_mini_cluster.cc:1428] Setting key cac9fd935455925d22e7ff975e8c5581
I20260925 02:35:32.279490 31294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da: Bootstrap starting.
I20260925 02:35:32.280117 31294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:32.280438 31294 log.cc:824] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:32.281196 31294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da: No bootstrap required, opened a new log
I20260925 02:35:32.283939 31294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c896ef7fde64b348c4987be488d98da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260925 02:35:32.284072 31294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:32.284094 31294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c896ef7fde64b348c4987be488d98da, State: Initialized, Role: FOLLOWER
I20260925 02:35:32.284229 31294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c896ef7fde64b348c4987be488d98da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260925 02:35:32.284297 31294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:32.284318 31294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:32.284389 31294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:35:32 dist-test-slave-q8mg krb5kdc[31213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:35:32.285004 31294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c896ef7fde64b348c4987be488d98da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260925 02:35:32.285091 31294 leader_election.cc:302] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c896ef7fde64b348c4987be488d98da; no voters:
I20260925 02:35:32.285286 31294 leader_election.cc:288] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:32.285352 31299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:32.285566 31299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [term 1 LEADER]: Becoming Leader. State: Replica: 2c896ef7fde64b348c4987be488d98da, State: Running, Role: LEADER
I20260925 02:35:32.285583 31294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:32.285712 31299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c896ef7fde64b348c4987be488d98da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260925 02:35:32.286214 31300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c896ef7fde64b348c4987be488d98da. Latest consensus state: current_term: 1 leader_uuid: "2c896ef7fde64b348c4987be488d98da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c896ef7fde64b348c4987be488d98da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20260925 02:35:32.286373 31300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:32.286233 31301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c896ef7fde64b348c4987be488d98da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c896ef7fde64b348c4987be488d98da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20260925 02:35:32.286641 31301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:32.286651 31308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:32.287259 31308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:32.289142 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:32.277845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:56207 (local address 127.0.0.1:36245)
0925 02:35:32.278023 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:32.278031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:32.278061 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:35:32.278484 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:32.278491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:32.278540 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:32.278654 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:32.279498 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:32.280541 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:32.281537 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:32.281948 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:32.282034 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:32.284915 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:32.284920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:32.284920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:32.284933 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:32.287183 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:32.287792 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:32.287796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:32.287796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:32.287901 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:32.288205 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:32.288207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:32.288207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:32.288523 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:32.288679 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:32.288820 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 02:35:32.288942 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260925 02:35:32.290283 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:32.295439 31308 catalog_manager.cc:1347] Generated new cluster ID: d3672c0f8c214bb8a6d2cbc3a41cf0be
I20260925 02:35:32.295522 31308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:32.298972 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:32.291766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:58089 (local address 127.0.0.1:36245)
0925 02:35:32.291807 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:32.291809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:32.291932 (+ 123us) server_negotiation.cc:407] Connection header received
0925 02:35:32.291987 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:32.291988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:32.291997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:32.292042 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:32.292678 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:32.293221 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:32.294293 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:32.294509 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:32.294559 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:32.295946 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:32.295948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:32.295948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:32.295955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:32.297600 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:32.298050 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:32.298051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:32.298051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:32.298107 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:32.298362 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:32.298363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:32.298364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:32.298488 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:32.298575 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:32.298821 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:35:32.298884 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:35:32.300709 31308 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:32.301460 31308 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:32.318789 31308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c896ef7fde64b348c4987be488d98da: Generated new TSK 0
I20260925 02:35:32.319032 31308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:32.323693 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31226
W20260925 02:35:32.328024 31211 connection.cc:535] client connection to 127.0.0.1:36245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36245): Connection reset by peer (error 104)
2026-09-25T02:35:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:32 dist-test-slave-q8mg krb5kdc[31325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:32 dist-test-slave-q8mg krb5kdc[31325](info): set up 2 sockets
Sep 25 02:35:32 dist-test-slave-q8mg krb5kdc[31325](info): commencing operation
krb5kdc: starting...
W20260925 02:35:33.367413 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:33 dist-test-slave-q8mg krb5kdc[31325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:33.488425 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:33.571401 31338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:33.571612 31338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:33.571636 31338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:33.572826 31338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:33.572937 31338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:33.572999 31338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:33.573021 31338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:33.573042 31338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:33.574592 31338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:33.574900 31338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:33.575145 31338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:33.580471 31346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:33.580495 31343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:33.580575 31338 server_base.cc:1034] running on GCE node
W20260925 02:35:33.580528 31344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:33.581046 31338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:33.581324 31338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:33.582491 31338 hybrid_clock.cc:648] HybridClock initialized: now 1790303733582481 us; error 32 us; skew 500 ppm
Sep 25 02:35:33 dist-test-slave-q8mg krb5kdc[31325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:33.585181 31338 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:33.585618 31338 webserver.cc:460] Webserver started at http://127.0.0.1:42209/ using document root <none> and password file <none>
I20260925 02:35:33.585836 31338 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:33.585878 31338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:33.585996 31338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:33.587082 31338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "09171332d05747d89669c84640d0a93a"
format_stamp: "Formatted at 2026-09-25 02:35:33 on dist-test-slave-q8mg"
server_key: "0b0fffbe0eed824038fd9ee06dfad16c"
server_key_iv: "c576d50a5ae1a9266cb3091c8b496fe4"
server_key_version: "encryptionkey@0"
I20260925 02:35:33.587390 31338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "09171332d05747d89669c84640d0a93a"
format_stamp: "Formatted at 2026-09-25 02:35:33 on dist-test-slave-q8mg"
server_key: "0b0fffbe0eed824038fd9ee06dfad16c"
server_key_iv: "c576d50a5ae1a9266cb3091c8b496fe4"
server_key_version: "encryptionkey@0"
I20260925 02:35:33.588681 31338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:33.589497 31353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:33.589676 31338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:33.589761 31338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "09171332d05747d89669c84640d0a93a"
format_stamp: "Formatted at 2026-09-25 02:35:33 on dist-test-slave-q8mg"
server_key: "0b0fffbe0eed824038fd9ee06dfad16c"
server_key_iv: "c576d50a5ae1a9266cb3091c8b496fe4"
server_key_version: "encryptionkey@0"
I20260925 02:35:33.589818 31338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:33.604357 31338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:33.606644 31338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:33.606762 31338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:33.610894 31338 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35647
I20260925 02:35:33.610955 31406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35647 every 8 connection(s)
I20260925 02:35:33.611337 31338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:33.612423 31407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:33.613615 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31338
I20260925 02:35:33.613766 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:33.613966 10292 external_mini_cluster.cc:1428] Setting key 2125d59424c7a86a12d7b4ca47d0fb46
I20260925 02:35:33.615268 31407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a: Bootstrap starting.
I20260925 02:35:33.615919 31407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:33.616218 31407 log.cc:824] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:33.617020 31407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a: No bootstrap required, opened a new log
I20260925 02:35:33.619021 31407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09171332d05747d89669c84640d0a93a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } }
I20260925 02:35:33.619191 31407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:33.619246 31407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09171332d05747d89669c84640d0a93a, State: Initialized, Role: FOLLOWER
I20260925 02:35:33.619424 31407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09171332d05747d89669c84640d0a93a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } }
I20260925 02:35:33.619561 31407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:33.619612 31407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:33.619678 31407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:33.619835 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:33.614506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:44017 (local address 127.0.0.1:35647)
0925 02:35:33.614716 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:33.614729 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:33.614756 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:35:33.615175 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:33.615182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:33.615233 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:33.615391 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:33.615678 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:33.616690 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:33.617618 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:33.618053 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:33.618178 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:33.619001 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:33.619006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:33.619006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:33.619174 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:33.619206 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:33.619238 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:33.619346 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:33.619522 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:35:33.619559 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260925 02:35:33.620401 31407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09171332d05747d89669c84640d0a93a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } }
I20260925 02:35:33.620543 31407 leader_election.cc:302] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09171332d05747d89669c84640d0a93a; no voters:
I20260925 02:35:33.620692 31407 leader_election.cc:288] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:33.620728 31412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:33.620899 31412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [term 1 LEADER]: Becoming Leader. State: Replica: 09171332d05747d89669c84640d0a93a, State: Running, Role: LEADER
I20260925 02:35:33.620965 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:33.620996 31407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:33.621037 31412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09171332d05747d89669c84640d0a93a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } }
I20260925 02:35:33.621392 31412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09171332d05747d89669c84640d0a93a. Latest consensus state: current_term: 1 leader_uuid: "09171332d05747d89669c84640d0a93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09171332d05747d89669c84640d0a93a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } } }
I20260925 02:35:33.621399 31413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09171332d05747d89669c84640d0a93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09171332d05747d89669c84640d0a93a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } } }
I20260925 02:35:33.621515 31412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:33.621528 31413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:33.622547 31418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:33.623111 31418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:33.624974 31418 catalog_manager.cc:1347] Generated new cluster ID: af59915b645d4bd5884f704791915f7b
I20260925 02:35:33.625029 31418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:33.626271 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:33.622570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:41125 (local address 127.0.0.1:35647)
0925 02:35:33.622606 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:33.622608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:33.622759 (+ 151us) server_negotiation.cc:407] Connection header received
0925 02:35:33.622958 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:33.622959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:33.622971 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:33.623036 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:33.623188 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:33.623875 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:33.624803 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:33.625160 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:33.625230 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:33.626017 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:33.626019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:33.626019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:33.626024 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:33.626038 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:33.626042 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:33.626072 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:33.626182 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 02:35:33.626193 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:35:33.648093 31418 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:33.648659 31418 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:33.651913 31418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09171332d05747d89669c84640d0a93a: Generated new TSK 0
I20260925 02:35:33.652139 31418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:33.660513 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31338
W20260925 02:35:33.665581 31323 connection.cc:535] client connection to 127.0.0.1:35647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35647): Connection reset by peer (error 104)
2026-09-25T02:35:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:33 dist-test-slave-q8mg krb5kdc[31438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:33 dist-test-slave-q8mg krb5kdc[31438](info): set up 2 sockets
Sep 25 02:35:33 dist-test-slave-q8mg krb5kdc[31438](info): commencing operation
krb5kdc: starting...
W20260925 02:35:34.699499 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:34 dist-test-slave-q8mg krb5kdc[31438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:34.816279 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:34.899610 31451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:34.899883 31451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:34.899928 31451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:34.901118 31451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:34.901190 31451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:34.901213 31451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:34.901233 31451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:34.901262 31451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:34.902770 31451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:34.903061 31451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:34.903294 31451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:34.908180 31460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:34.908291 31457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:34.908195 31456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:34.908406 31451 server_base.cc:1034] running on GCE node
I20260925 02:35:34.908661 31451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:34.909016 31451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:34.910213 31451 hybrid_clock.cc:648] HybridClock initialized: now 1790303734910199 us; error 40 us; skew 500 ppm
Sep 25 02:35:34 dist-test-slave-q8mg krb5kdc[31438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:34.912837 31451 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:34.913340 31451 webserver.cc:460] Webserver started at http://127.0.0.1:34793/ using document root <none> and password file <none>
I20260925 02:35:34.913585 31451 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:34.913654 31451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:34.913826 31451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:34.914883 31451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "413ea29307db4803a7cfbd4d07c7b500"
format_stamp: "Formatted at 2026-09-25 02:35:34 on dist-test-slave-q8mg"
server_key: "2550ddfaf4284883bf1a902cbda61067"
server_key_iv: "082df7cbb055d94768c79621ccf9e20d"
server_key_version: "encryptionkey@0"
I20260925 02:35:34.915266 31451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "413ea29307db4803a7cfbd4d07c7b500"
format_stamp: "Formatted at 2026-09-25 02:35:34 on dist-test-slave-q8mg"
server_key: "2550ddfaf4284883bf1a902cbda61067"
server_key_iv: "082df7cbb055d94768c79621ccf9e20d"
server_key_version: "encryptionkey@0"
I20260925 02:35:34.916610 31451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:34.917476 31467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:34.917675 31451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:35:34.917779 31451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "413ea29307db4803a7cfbd4d07c7b500"
format_stamp: "Formatted at 2026-09-25 02:35:34 on dist-test-slave-q8mg"
server_key: "2550ddfaf4284883bf1a902cbda61067"
server_key_iv: "082df7cbb055d94768c79621ccf9e20d"
server_key_version: "encryptionkey@0"
I20260925 02:35:34.917869 31451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:34.921128 31451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:34.923038 31451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:34.923197 31451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:34.927248 31451 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45323
I20260925 02:35:34.927280 31519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45323 every 8 connection(s)
I20260925 02:35:34.927621 31451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:34.928637 31520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:34.930964 31520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500: Bootstrap starting.
I20260925 02:35:34.931000 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31451
I20260925 02:35:34.931144 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:34.931331 10292 external_mini_cluster.cc:1428] Setting key 0f7af7d0de0262a99530ba06978c3a4d
I20260925 02:35:34.931751 31520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:34.932176 31520 log.cc:824] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:34.932960 31520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500: No bootstrap required, opened a new log
I20260925 02:35:34.934194 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:34.931874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:59243 (local address 127.0.0.1:45323)
0925 02:35:34.932108 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:34.932117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:34.932156 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:35:34.932749 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:34.932758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:34.932795 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:34.932968 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:34.933706 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:34.933709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:34.933710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:34.933714 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:34.933766 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:34.933780 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:34.933882 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:34.933958 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 02:35:34.933999 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260925 02:35:34.935353 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:34.935436 31520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413ea29307db4803a7cfbd4d07c7b500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } }
I20260925 02:35:34.935587 31520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:34.935631 31520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413ea29307db4803a7cfbd4d07c7b500, State: Initialized, Role: FOLLOWER
I20260925 02:35:34.935741 31520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413ea29307db4803a7cfbd4d07c7b500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } }
I20260925 02:35:34.935823 31520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:34.935859 31520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:34.935911 31520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:34.936657 31520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413ea29307db4803a7cfbd4d07c7b500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } }
I20260925 02:35:34.936772 31520 leader_election.cc:302] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 413ea29307db4803a7cfbd4d07c7b500; no voters:
I20260925 02:35:34.936983 31520 leader_election.cc:288] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:34.937036 31530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:34.937167 31530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [term 1 LEADER]: Becoming Leader. State: Replica: 413ea29307db4803a7cfbd4d07c7b500, State: Running, Role: LEADER
I20260925 02:35:34.937273 31520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:34.937309 31530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413ea29307db4803a7cfbd4d07c7b500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } }
I20260925 02:35:34.937707 31533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 413ea29307db4803a7cfbd4d07c7b500. Latest consensus state: current_term: 1 leader_uuid: "413ea29307db4803a7cfbd4d07c7b500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413ea29307db4803a7cfbd4d07c7b500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } } }
I20260925 02:35:34.937721 31532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "413ea29307db4803a7cfbd4d07c7b500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413ea29307db4803a7cfbd4d07c7b500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } } }
I20260925 02:35:34.937861 31533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:34.937876 31532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500 [sys.catalog]: This master's current role is: LEADER
W20260925 02:35:34.939073 31547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:34.939153 31547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:34.939556 31540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:34.939774 31540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:34.939949 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:34.936969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:51467 (local address 127.0.0.1:45323)
0925 02:35:34.937008 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:34.937009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:34.938842 (+ 1833us) server_negotiation.cc:407] Connection header received
0925 02:35:34.938951 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:34.938953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:34.938958 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:34.939008 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:34.939615 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:34.939616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:34.939617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:34.939622 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:34.939637 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:34.939642 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:34.939693 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:34.939770 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 02:35:34.939783 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:35:34.941501 31540 catalog_manager.cc:1347] Generated new cluster ID: a7da67db4a2d43e185ba896e7e54aad5
I20260925 02:35:34.941550 31540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:34.968145 31540 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:34.968359 31540 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:34.978307 31540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 413ea29307db4803a7cfbd4d07c7b500: Generated new TSK 0
I20260925 02:35:34.978550 31540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:34.983093 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31451
2026-09-25T02:35:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:35 dist-test-slave-q8mg krb5kdc[31552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:35 dist-test-slave-q8mg krb5kdc[31552](info): set up 2 sockets
Sep 25 02:35:35 dist-test-slave-q8mg krb5kdc[31552](info): commencing operation
krb5kdc: starting...
W20260925 02:35:36.043478 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:36 dist-test-slave-q8mg krb5kdc[31552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:36.164074 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:36.248526 31565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:36.248759 31565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:36.248798 31565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:36.249993 31565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:36.250146 31565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:36.250239 31565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:36.250273 31565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:36.250303 31565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:36.252022 31565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:36.252305 31565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:36.252547 31565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:36.257480 31571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:36.257622 31573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:36.257658 31570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:36.257848 31565 server_base.cc:1034] running on GCE node
I20260925 02:35:36.258080 31565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:36.258379 31565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:36.259543 31565 hybrid_clock.cc:648] HybridClock initialized: now 1790303736259532 us; error 26 us; skew 500 ppm
Sep 25 02:35:36 dist-test-slave-q8mg krb5kdc[31552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:36.262045 31565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:36.262511 31565 webserver.cc:460] Webserver started at http://127.0.0.1:34113/ using document root <none> and password file <none>
I20260925 02:35:36.262718 31565 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:36.262759 31565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:36.262848 31565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:36.263862 31565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "45a206bb44b4451ebb402a86c04e6610"
format_stamp: "Formatted at 2026-09-25 02:35:36 on dist-test-slave-q8mg"
server_key: "ffc3d586f8abd801399f6033c21ac02f"
server_key_iv: "f2848669db5ad2a70f0cdccb6cfc8152"
server_key_version: "encryptionkey@0"
I20260925 02:35:36.264155 31565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "45a206bb44b4451ebb402a86c04e6610"
format_stamp: "Formatted at 2026-09-25 02:35:36 on dist-test-slave-q8mg"
server_key: "ffc3d586f8abd801399f6033c21ac02f"
server_key_iv: "f2848669db5ad2a70f0cdccb6cfc8152"
server_key_version: "encryptionkey@0"
I20260925 02:35:36.265524 31565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:36.266242 31580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:36.266438 31565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:36.266526 31565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "45a206bb44b4451ebb402a86c04e6610"
format_stamp: "Formatted at 2026-09-25 02:35:36 on dist-test-slave-q8mg"
server_key: "ffc3d586f8abd801399f6033c21ac02f"
server_key_iv: "f2848669db5ad2a70f0cdccb6cfc8152"
server_key_version: "encryptionkey@0"
I20260925 02:35:36.266583 31565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:36.292109 31565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:36.294574 31565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:36.294695 31565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:36.298866 31565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38909
I20260925 02:35:36.298941 31633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38909 every 8 connection(s)
I20260925 02:35:36.299283 31565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:36.300225 31634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:36.302613 31634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610: Bootstrap starting.
I20260925 02:35:36.303211 31634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:36.303495 31634 log.cc:824] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:36.304174 31634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610: No bootstrap required, opened a new log
I20260925 02:35:36.305810 31634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a206bb44b4451ebb402a86c04e6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } }
I20260925 02:35:36.305981 31634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:36.306041 31634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a206bb44b4451ebb402a86c04e6610, State: Initialized, Role: FOLLOWER
I20260925 02:35:36.306202 31634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a206bb44b4451ebb402a86c04e6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } }
I20260925 02:35:36.306281 31634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:36.306325 31634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:36.306375 31634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:36.306888 31634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a206bb44b4451ebb402a86c04e6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } }
I20260925 02:35:36.307001 31634 leader_election.cc:302] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45a206bb44b4451ebb402a86c04e6610; no voters:
I20260925 02:35:36.307178 31634 leader_election.cc:288] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:36.307224 31637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:36.307431 31634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:36.307555 31637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [term 1 LEADER]: Becoming Leader. State: Replica: 45a206bb44b4451ebb402a86c04e6610, State: Running, Role: LEADER
I20260925 02:35:36.307651 31637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a206bb44b4451ebb402a86c04e6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } }
I20260925 02:35:36.308229 31639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45a206bb44b4451ebb402a86c04e6610. Latest consensus state: current_term: 1 leader_uuid: "45a206bb44b4451ebb402a86c04e6610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a206bb44b4451ebb402a86c04e6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } } }
I20260925 02:35:36.308319 31639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:36.308585 31646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:36.308758 31638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45a206bb44b4451ebb402a86c04e6610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a206bb44b4451ebb402a86c04e6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } } }
I20260925 02:35:36.308851 31638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:36.309072 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31565
I20260925 02:35:36.309211 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:36.309302 31646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:36.309423 10292 external_mini_cluster.cc:1428] Setting key d5e9ffacd281f22b13b54a19e830ea05
I20260925 02:35:36.311498 31646 catalog_manager.cc:1347] Generated new cluster ID: cb413ed166314c25acdb638121167d5c
I20260925 02:35:36.311554 31646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:35:36 dist-test-slave-q8mg krb5kdc[31552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:35:36.321265 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:36.310105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:36581 (local address 127.0.0.1:38909)
0925 02:35:36.310284 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:36.310294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:36.310326 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:35:36.310463 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:36.310470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:36.310528 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:36.313000 (+ 2472us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:36.313084 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:36.314028 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:36.314838 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:36.315262 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:36.315393 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:36.317837 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:36.317842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:36.317842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:36.317863 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:36.319572 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:36.319988 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:36.319992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:36.319994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:36.320086 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:36.320324 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:36.320325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:36.320325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:36.320586 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:36.320773 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:36.320900 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 02:35:36.321034 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260925 02:35:36.322374 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:36.331216 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:36.323738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:60211 (local address 127.0.0.1:38909)
0925 02:35:36.323781 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:36.323783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:36.325074 (+ 1291us) server_negotiation.cc:407] Connection header received
0925 02:35:36.325149 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:36.325150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:36.325160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:36.325203 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:36.325857 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:36.326344 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:36.327056 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:36.327282 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:36.327361 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:36.328599 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:36.328600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:36.328601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:36.328608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:36.330019 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:36.330396 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:36.330397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:36.330397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:36.330454 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:36.330691 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:36.330692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:36.330692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:36.330808 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:36.330917 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:36.331045 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:35:36.331108 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:35:36.343284 31646 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:36.344027 31646 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:36.347419 31646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45a206bb44b4451ebb402a86c04e6610: Generated new TSK 0
I20260925 02:35:36.347615 31646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:36.351476 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31565
W20260925 02:35:36.356178 31550 connection.cc:535] client connection to 127.0.0.1:38909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38909): Connection reset by peer (error 104)
2026-09-25T02:35:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:36 dist-test-slave-q8mg krb5kdc[31665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:36 dist-test-slave-q8mg krb5kdc[31665](info): set up 2 sockets
Sep 25 02:35:36 dist-test-slave-q8mg krb5kdc[31665](info): commencing operation
krb5kdc: starting...
W20260925 02:35:37.391419 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:37 dist-test-slave-q8mg krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:37.512455 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:37.595345 31678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:37.595562 31678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:37.595585 31678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:37.596788 31678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:37.596832 31678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:37.596845 31678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:37.596858 31678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:37.596922 31678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:37.598397 31678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:37.598658 31678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:37.598909 31678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:37.603953 31684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:37.603961 31678 server_base.cc:1034] running on GCE node
W20260925 02:35:37.603929 31683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:37.603929 31686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:37.604506 31678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:37.604768 31678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:37.605913 31678 hybrid_clock.cc:648] HybridClock initialized: now 1790303737605905 us; error 39 us; skew 500 ppm
Sep 25 02:35:37 dist-test-slave-q8mg krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:37.616137 31678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:37.616739 31678 webserver.cc:460] Webserver started at http://127.0.0.1:42585/ using document root <none> and password file <none>
I20260925 02:35:37.617040 31678 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:37.617092 31678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:37.617254 31678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:37.618428 31678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "2a72594346034e29b3a1152331a95123"
format_stamp: "Formatted at 2026-09-25 02:35:37 on dist-test-slave-q8mg"
server_key: "ad40cda581e6675d02764f6a7d823224"
server_key_iv: "576cce036a60f388c0742d7356d1f509"
server_key_version: "encryptionkey@0"
I20260925 02:35:37.618799 31678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "2a72594346034e29b3a1152331a95123"
format_stamp: "Formatted at 2026-09-25 02:35:37 on dist-test-slave-q8mg"
server_key: "ad40cda581e6675d02764f6a7d823224"
server_key_iv: "576cce036a60f388c0742d7356d1f509"
server_key_version: "encryptionkey@0"
I20260925 02:35:37.620211 31678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:37.621078 31693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:37.621285 31678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:37.621394 31678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "2a72594346034e29b3a1152331a95123"
format_stamp: "Formatted at 2026-09-25 02:35:37 on dist-test-slave-q8mg"
server_key: "ad40cda581e6675d02764f6a7d823224"
server_key_iv: "576cce036a60f388c0742d7356d1f509"
server_key_version: "encryptionkey@0"
I20260925 02:35:37.621472 31678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:37.635831 31678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:37.637758 31678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:37.637925 31678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:37.642320 31678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42183
I20260925 02:35:37.642380 31746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42183 every 8 connection(s)
I20260925 02:35:37.642751 31678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:37.643630 31747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:37.646024 31747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123: Bootstrap starting.
I20260925 02:35:37.646625 31747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:37.646909 31747 log.cc:824] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:37.647472 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31678
I20260925 02:35:37.647601 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:37.647715 31747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123: No bootstrap required, opened a new log
I20260925 02:35:37.647787 10292 external_mini_cluster.cc:1428] Setting key 876ae78fabcc4d77285c654057a8180e
I20260925 02:35:37.650516 31747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a72594346034e29b3a1152331a95123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260925 02:35:37.650687 31747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:37.650727 31747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a72594346034e29b3a1152331a95123, State: Initialized, Role: FOLLOWER
I20260925 02:35:37.650877 31747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a72594346034e29b3a1152331a95123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260925 02:35:37.650972 31747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:37.651003 31747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:37.651057 31747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:37.651796 31747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a72594346034e29b3a1152331a95123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260925 02:35:37.651891 31747 leader_election.cc:302] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a72594346034e29b3a1152331a95123; no voters:
I20260925 02:35:37.652078 31747 leader_election.cc:288] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:37.652101 31752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:37.652217 31752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [term 1 LEADER]: Becoming Leader. State: Replica: 2a72594346034e29b3a1152331a95123, State: Running, Role: LEADER
I20260925 02:35:37.652360 31747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:37.652372 31752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a72594346034e29b3a1152331a95123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260925 02:35:37.652833 31753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a72594346034e29b3a1152331a95123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a72594346034e29b3a1152331a95123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } } }
I20260925 02:35:37.653117 31753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:37.653036 31754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a72594346034e29b3a1152331a95123. Latest consensus state: current_term: 1 leader_uuid: "2a72594346034e29b3a1152331a95123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a72594346034e29b3a1152331a95123" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } } }
I20260925 02:35:37.653676 31754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123 [sys.catalog]: This master's current role is: LEADER
W20260925 02:35:37.654278 31768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:37.654343 31768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:37.654421 31761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:37.654610 31761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:37.654770 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:37.648355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:38831 (local address 127.0.0.1:42183)
0925 02:35:37.648548 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:37.648556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:37.648585 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:35:37.649128 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:37.649135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:37.649188 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:37.649309 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:37.649528 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:37.650573 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:37.651795 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:37.652422 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:37.652862 (+ 440us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:37.653838 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:37.653842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:37.653843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:37.654064 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:37.654111 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:37.654126 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:37.654440 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:37.654488 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 02:35:37.654516 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260925 02:35:37.655757 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:37.657624 31761 catalog_manager.cc:1347] Generated new cluster ID: 1fa58a74520f46a285226076661d96e4
I20260925 02:35:37.657682 31761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:37.660064 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:37.657268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:41585 (local address 127.0.0.1:42183)
0925 02:35:37.657304 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:37.657306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:37.657315 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:37.657361 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:37.657362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:37.657382 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:37.657449 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:37.657644 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:37.658135 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:37.658856 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:37.659081 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:37.659157 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:37.659809 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:37.659810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:37.659810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:37.659814 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:37.659827 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:37.659831 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:37.659882 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:37.659962 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 02:35:37.659971 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:35:37.686187 31761 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:37.686767 31761 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:37.695631 31761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a72594346034e29b3a1152331a95123: Generated new TSK 0
I20260925 02:35:37.695840 31761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:37.702677 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31678
W20260925 02:35:37.707430 31663 connection.cc:535] client connection to 127.0.0.1:42183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42183): Connection reset by peer (error 104)
2026-09-25T02:35:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:37 dist-test-slave-q8mg krb5kdc[31779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:37 dist-test-slave-q8mg krb5kdc[31779](info): set up 2 sockets
Sep 25 02:35:37 dist-test-slave-q8mg krb5kdc[31779](info): commencing operation
krb5kdc: starting...
W20260925 02:35:38.743533 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:38 dist-test-slave-q8mg krb5kdc[31779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:35:38.866283 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:38.949258 31792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:38.949525 31792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:38.949565 31792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:38.950742 31792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:38.950815 31792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:38.950839 31792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:38.950868 31792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:38.950901 31792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:38.952397 31792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:38.952696 31792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:38.952951 31792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:38.958405 31800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:38.958446 31798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:38.958426 31797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:38.958674 31792 server_base.cc:1034] running on GCE node
I20260925 02:35:38.958932 31792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:38.959233 31792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:38.960382 31792 hybrid_clock.cc:648] HybridClock initialized: now 1790303738960363 us; error 41 us; skew 500 ppm
Sep 25 02:35:38 dist-test-slave-q8mg krb5kdc[31779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:38.963032 31792 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:35:38.963480 31792 webserver.cc:460] Webserver started at http://127.0.0.1:45777/ using document root <none> and password file <none>
I20260925 02:35:38.963713 31792 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:38.963781 31792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:38.963938 31792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:38.965080 31792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "ee485b8472864e56ae2e46b017cb8533"
format_stamp: "Formatted at 2026-09-25 02:35:38 on dist-test-slave-q8mg"
server_key: "5e6232d77b99d2bef36bfaf222eb8bdc"
server_key_iv: "5971ae2526a2483327f31327751b5605"
server_key_version: "encryptionkey@0"
I20260925 02:35:38.965437 31792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "ee485b8472864e56ae2e46b017cb8533"
format_stamp: "Formatted at 2026-09-25 02:35:38 on dist-test-slave-q8mg"
server_key: "5e6232d77b99d2bef36bfaf222eb8bdc"
server_key_iv: "5971ae2526a2483327f31327751b5605"
server_key_version: "encryptionkey@0"
I20260925 02:35:38.966780 31792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:35:38.967561 31807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:38.967736 31792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:38.967845 31792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "ee485b8472864e56ae2e46b017cb8533"
format_stamp: "Formatted at 2026-09-25 02:35:38 on dist-test-slave-q8mg"
server_key: "5e6232d77b99d2bef36bfaf222eb8bdc"
server_key_iv: "5971ae2526a2483327f31327751b5605"
server_key_version: "encryptionkey@0"
I20260925 02:35:38.967938 31792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:38.971300 31792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:38.973413 31792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:38.973565 31792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:38.977525 31792 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39371
I20260925 02:35:38.977574 31859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39371 every 8 connection(s)
I20260925 02:35:38.977934 31792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:38.978966 31860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:38.981175 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31792
I20260925 02:35:38.981300 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:38.981491 10292 external_mini_cluster.cc:1428] Setting key 744818fd51b3f894d941d0d808c1a1f6
I20260925 02:35:38.981611 31860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533: Bootstrap starting.
I20260925 02:35:38.982393 31860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:38.982734 31860 log.cc:824] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:38.983523 31860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533: No bootstrap required, opened a new log
I20260925 02:35:38.984179 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:38.982074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:33277 (local address 127.0.0.1:39371)
0925 02:35:38.982278 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:38.982294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:38.982320 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:35:38.982837 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:38.982846 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:38.982881 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:38.983027 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:38.983717 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:38.983721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:38.983721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:38.983725 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:38.983782 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:38.983793 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:38.983885 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:38.983963 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:35:38.984011 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260925 02:35:38.985334 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:38.986207 31860 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee485b8472864e56ae2e46b017cb8533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260925 02:35:38.986392 31860 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:38.986444 31860 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee485b8472864e56ae2e46b017cb8533, State: Initialized, Role: FOLLOWER
I20260925 02:35:38.986594 31860 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee485b8472864e56ae2e46b017cb8533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260925 02:35:38.986712 31860 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:38.986749 31860 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:38.986801 31860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:38.987454 31860 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee485b8472864e56ae2e46b017cb8533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260925 02:35:38.987587 31860 leader_election.cc:302] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee485b8472864e56ae2e46b017cb8533; no voters:
I20260925 02:35:38.987763 31860 leader_election.cc:288] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:38.987787 31870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:38.988032 31860 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:38.988164 31870 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [term 1 LEADER]: Becoming Leader. State: Replica: ee485b8472864e56ae2e46b017cb8533, State: Running, Role: LEADER
I20260925 02:35:38.988291 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:38.987029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:33433 (local address 127.0.0.1:39371)
0925 02:35:38.987066 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:38.987068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:38.987199 (+ 131us) server_negotiation.cc:407] Connection header received
0925 02:35:38.987274 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:38.987276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:38.987281 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:38.987329 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:38.987961 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:38.987962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:38.987963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:38.987967 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:38.987981 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:38.987985 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:38.988033 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:38.988176 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:35:38.988189 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 02:35:38.988394 31870 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee485b8472864e56ae2e46b017cb8533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260925 02:35:38.988752 31871 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee485b8472864e56ae2e46b017cb8533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee485b8472864e56ae2e46b017cb8533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } } }
I20260925 02:35:38.988830 31871 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:38.989004 31872 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee485b8472864e56ae2e46b017cb8533. Latest consensus state: current_term: 1 leader_uuid: "ee485b8472864e56ae2e46b017cb8533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee485b8472864e56ae2e46b017cb8533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } } }
I20260925 02:35:38.989064 31879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:38.989082 31872 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:38.989677 31879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:38.993928 31879 catalog_manager.cc:1347] Generated new cluster ID: f486a1a009d847d2b642a0627d448f2e
I20260925 02:35:38.994015 31879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:39.001052 31879 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:39.001215 31879 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:39.004491 31879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee485b8472864e56ae2e46b017cb8533: Generated new TSK 0
I20260925 02:35:39.004832 31879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:39.011626 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31792
2026-09-25T02:35:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:39 dist-test-slave-q8mg krb5kdc[31891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:39 dist-test-slave-q8mg krb5kdc[31891](info): set up 2 sockets
Sep 25 02:35:39 dist-test-slave-q8mg krb5kdc[31891](info): commencing operation
krb5kdc: starting...
W20260925 02:35:40.051550 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:40 dist-test-slave-q8mg krb5kdc[31891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:40.171944 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40463
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:40463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:40.253330 31904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:40.253540 31904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:40.253563 31904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:40.254652 31904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:40.254689 31904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:40.254702 31904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:40.254715 31904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:40.254727 31904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:40.256157 31904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:40463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:40.256443 31904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:40.256723 31904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:40.262372 31910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:40.262394 31904 server_base.cc:1034] running on GCE node
W20260925 02:35:40.262360 31911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:40.262344 31913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:40.262719 31904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:40.262975 31904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:40.264122 31904 hybrid_clock.cc:648] HybridClock initialized: now 1790303740264106 us; error 32 us; skew 500 ppm
Sep 25 02:35:40 dist-test-slave-q8mg krb5kdc[31891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303740, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:40.266605 31904 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:40.267091 31904 webserver.cc:460] Webserver started at http://127.10.13.62:37407/ using document root <none> and password file <none>
I20260925 02:35:40.267328 31904 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:40.267629 31904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:40.267879 31904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:40.269080 31904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "f2151ce2acbd461d8e624ee35a5cf8b3"
format_stamp: "Formatted at 2026-09-25 02:35:40 on dist-test-slave-q8mg"
server_key: "e3724abfcba84741dbc8c57963e8da2a"
server_key_iv: "1a7055bb4914cd3d56cc0446685dc8a1"
server_key_version: "encryptionkey@0"
I20260925 02:35:40.269484 31904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "f2151ce2acbd461d8e624ee35a5cf8b3"
format_stamp: "Formatted at 2026-09-25 02:35:40 on dist-test-slave-q8mg"
server_key: "e3724abfcba84741dbc8c57963e8da2a"
server_key_iv: "1a7055bb4914cd3d56cc0446685dc8a1"
server_key_version: "encryptionkey@0"
I20260925 02:35:40.270892 31904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:35:40.271683 31920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:40.271891 31904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:40.272007 31904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "f2151ce2acbd461d8e624ee35a5cf8b3"
format_stamp: "Formatted at 2026-09-25 02:35:40 on dist-test-slave-q8mg"
server_key: "e3724abfcba84741dbc8c57963e8da2a"
server_key_iv: "1a7055bb4914cd3d56cc0446685dc8a1"
server_key_version: "encryptionkey@0"
I20260925 02:35:40.272085 31904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:40.314468 31904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:40.316375 31904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:40.316545 31904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:40.320611 31904 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:40463
I20260925 02:35:40.320670 31972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:40463 every 8 connection(s)
I20260925 02:35:40.321067 31904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:40.321959 31973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:40.324213 31973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3: Bootstrap starting.
I20260925 02:35:40.324759 31973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:40.325033 31973 log.cc:824] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:40.325683 31973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3: No bootstrap required, opened a new log
I20260925 02:35:40.327353 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 31904
I20260925 02:35:40.327488 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:40.327452 31973 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2151ce2acbd461d8e624ee35a5cf8b3" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40463 } }
I20260925 02:35:40.327636 31973 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:40.327677 10292 external_mini_cluster.cc:1428] Setting key c9586095e1826d6bf1e2ef5349c2f000
I20260925 02:35:40.327673 31973 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2151ce2acbd461d8e624ee35a5cf8b3, State: Initialized, Role: FOLLOWER
I20260925 02:35:40.327826 31973 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2151ce2acbd461d8e624ee35a5cf8b3" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40463 } }
I20260925 02:35:40.327914 31973 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:40.327978 31973 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:40.328029 31973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:40.329087 31973 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2151ce2acbd461d8e624ee35a5cf8b3" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40463 } }
I20260925 02:35:40.329202 31973 leader_election.cc:302] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2151ce2acbd461d8e624ee35a5cf8b3; no voters:
I20260925 02:35:40.329424 31973 leader_election.cc:288] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:40.329457 31978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:40.329584 31978 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [term 1 LEADER]: Becoming Leader. State: Replica: f2151ce2acbd461d8e624ee35a5cf8b3, State: Running, Role: LEADER
I20260925 02:35:40.329679 31978 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2151ce2acbd461d8e624ee35a5cf8b3" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40463 } }
I20260925 02:35:40.329711 31973 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:40.331399 31978 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2151ce2acbd461d8e624ee35a5cf8b3. Latest consensus state: current_term: 1 leader_uuid: "f2151ce2acbd461d8e624ee35a5cf8b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2151ce2acbd461d8e624ee35a5cf8b3" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40463 } } }
I20260925 02:35:40.331547 31978 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:40.331574 31979 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2151ce2acbd461d8e624ee35a5cf8b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2151ce2acbd461d8e624ee35a5cf8b3" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40463 } } }
I20260925 02:35:40.331668 31979 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3 [sys.catalog]: This master's current role is: LEADER
W20260925 02:35:40.332405 31993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:40.332476 31993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:40.332572 31992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:40.332779 31992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:40.334389 31992 catalog_manager.cc:1347] Generated new cluster ID: bfdd521bd60c4bd1945784d92241aac9
I20260925 02:35:40.334447 31992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:35:40 dist-test-slave-q8mg krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:40.339838 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:40.328456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46953 (local address 127.10.13.62:40463)
0925 02:35:40.328653 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:40.328661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:40.328687 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:35:40.328781 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:40.328786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:40.328827 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:40.328994 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:40.329893 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:40.331231 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:40.332274 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:40.332865 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:40.333011 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:40.336095 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:40.336099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:40.336099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:40.336111 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:40.338079 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:40.338558 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:40.338562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:40.338562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:40.338672 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:40.338933 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:40.338934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:40.338934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:40.339216 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:40.339358 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:40.339548 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 02:35:40.339644 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260925 02:35:40.341003 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:40.346518 31992 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:40.347764 31992 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:40.349442 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:40.342410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55501 (local address 127.10.13.62:40463)
0925 02:35:40.342441 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:40.342443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:40.342627 (+ 184us) server_negotiation.cc:407] Connection header received
0925 02:35:40.342709 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:40.342710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:40.342718 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:40.342744 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:40.343366 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:40.344086 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:40.345104 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:40.345411 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:40.345448 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:40.346909 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:40.346911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:40.346911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:40.346919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:40.348234 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:40.348601 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:40.348603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:40.348603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:40.348663 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:40.348917 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:40.348918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:40.348918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:40.349033 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:40.349156 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:40.349280 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 02:35:40.349342 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 02:35:40.355764 31992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2151ce2acbd461d8e624ee35a5cf8b3: Generated new TSK 0
I20260925 02:35:40.355978 31992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:40.361692 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 31904
W20260925 02:35:40.366425 31889 connection.cc:535] client connection to 127.10.13.62:40463 recv error: Network error: failed to read from TLS socket (remote: 127.10.13.62:40463): Connection reset by peer (error 104)
2026-09-25T02:35:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:40 dist-test-slave-q8mg krb5kdc[32004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:40 dist-test-slave-q8mg krb5kdc[32004](info): set up 2 sockets
Sep 25 02:35:40 dist-test-slave-q8mg krb5kdc[32004](info): commencing operation
krb5kdc: starting...
W20260925 02:35:41.415437 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:41 dist-test-slave-q8mg krb5kdc[32004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:41.539552 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:33001
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:33001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:41.622709 32017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:41.622948 32017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:41.622972 32017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:41.624245 32017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:41.624292 32017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:41.624307 32017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:41.624320 32017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:41.624332 32017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:41.625913 32017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:33001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:33001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:41.626192 32017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:41.626433 32017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:41.631541 32022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:41.631524 32025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:41.631753 32017 server_base.cc:1034] running on GCE node
W20260925 02:35:41.631524 32023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:41.632059 32017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:41.632362 32017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:41.633507 32017 hybrid_clock.cc:648] HybridClock initialized: now 1790303741633489 us; error 39 us; skew 500 ppm
Sep 25 02:35:41 dist-test-slave-q8mg krb5kdc[32004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303741, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:41.636049 32017 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:41.636479 32017 webserver.cc:460] Webserver started at http://127.10.13.62:36125/ using document root <none> and password file <none>
I20260925 02:35:41.636708 32017 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:41.636760 32017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:41.636957 32017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:41.638017 32017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "500be56ac103467b8256056bca31228c"
format_stamp: "Formatted at 2026-09-25 02:35:41 on dist-test-slave-q8mg"
server_key: "38bc76c2b43ed947eb7821d999466b6e"
server_key_iv: "d6c00c3adef25678432a41cafe9f1934"
server_key_version: "encryptionkey@0"
I20260925 02:35:41.638381 32017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "500be56ac103467b8256056bca31228c"
format_stamp: "Formatted at 2026-09-25 02:35:41 on dist-test-slave-q8mg"
server_key: "38bc76c2b43ed947eb7821d999466b6e"
server_key_iv: "d6c00c3adef25678432a41cafe9f1934"
server_key_version: "encryptionkey@0"
I20260925 02:35:41.639758 32017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:41.640470 32032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:41.640653 32017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:35:41.640806 32017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "500be56ac103467b8256056bca31228c"
format_stamp: "Formatted at 2026-09-25 02:35:41 on dist-test-slave-q8mg"
server_key: "38bc76c2b43ed947eb7821d999466b6e"
server_key_iv: "d6c00c3adef25678432a41cafe9f1934"
server_key_version: "encryptionkey@0"
I20260925 02:35:41.640910 32017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:41.653539 32017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:41.655519 32017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:41.655694 32017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:41.659893 32017 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:33001
I20260925 02:35:41.659941 32084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:33001 every 8 connection(s)
I20260925 02:35:41.660318 32017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:41.661499 32085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:41.663857 32085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c: Bootstrap starting.
I20260925 02:35:41.664414 32085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:41.664692 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 32017
I20260925 02:35:41.664772 32085 log.cc:824] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:41.664824 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:41.665076 10292 external_mini_cluster.cc:1428] Setting key 12965ce89e14f36dc1520bf3b36c4144
I20260925 02:35:41.665709 32085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c: No bootstrap required, opened a new log
I20260925 02:35:41.668296 32085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500be56ac103467b8256056bca31228c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33001 } }
I20260925 02:35:41.668465 32085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:41.668498 32085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500be56ac103467b8256056bca31228c, State: Initialized, Role: FOLLOWER
I20260925 02:35:41.668615 32085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500be56ac103467b8256056bca31228c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33001 } }
I20260925 02:35:41.668691 32085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:41.668751 32085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:41.668809 32085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:41.669433 32085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500be56ac103467b8256056bca31228c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33001 } }
I20260925 02:35:41.669551 32085 leader_election.cc:302] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500be56ac103467b8256056bca31228c; no voters:
I20260925 02:35:41.669732 32085 leader_election.cc:288] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:41.669764 32090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:41.669903 32090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [term 1 LEADER]: Becoming Leader. State: Replica: 500be56ac103467b8256056bca31228c, State: Running, Role: LEADER
I20260925 02:35:41.670042 32085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:41.670063 32090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500be56ac103467b8256056bca31228c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33001 } }
I20260925 02:35:41.670463 32092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500be56ac103467b8256056bca31228c. Latest consensus state: current_term: 1 leader_uuid: "500be56ac103467b8256056bca31228c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500be56ac103467b8256056bca31228c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33001 } } }
I20260925 02:35:41.670562 32092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:41.670785 32096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:41.671146 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:41.665638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43847 (local address 127.10.13.62:33001)
0925 02:35:41.665815 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:41.665823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:41.665865 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:35:41.666268 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:41.666274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:41.666320 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:41.666435 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:41.666709 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:41.667582 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:41.668808 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:41.669317 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:41.669408 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:41.670239 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:41.670244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:41.670244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:41.670473 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:41.670517 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:41.670534 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:41.670621 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:41.670907 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 02:35:41.670945 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260925 02:35:41.671317 32096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:41.671692 32091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500be56ac103467b8256056bca31228c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500be56ac103467b8256056bca31228c" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33001 } } }
I20260925 02:35:41.672166 32091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:41.672262 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:41.673648 32096 catalog_manager.cc:1347] Generated new cluster ID: a98f4a7c976a45efad37cab1e1cf0216
I20260925 02:35:41.673718 32096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:41.677222 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:41.673933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56887 (local address 127.10.13.62:33001)
0925 02:35:41.673962 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:41.673964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:41.673972 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:35:41.673992 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:41.673992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:41.674000 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:41.674054 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:41.674192 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:41.674897 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:41.675915 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:41.676210 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:41.676267 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:41.676967 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:41.676968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:41.676968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:41.676973 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:41.676987 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:41.676991 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:41.677049 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:41.677145 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 02:35:41.677154 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 02:35:41.708570 32096 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:41.709211 32096 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:41.713393 32096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500be56ac103467b8256056bca31228c: Generated new TSK 0
I20260925 02:35:41.713599 32096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:41.717135 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 32017
W20260925 02:35:41.722144 32002 connection.cc:535] client connection to 127.10.13.62:33001 recv error: Network error: failed to read from TLS socket (remote: 127.10.13.62:33001): Connection reset by peer (error 104)
2026-09-25T02:35:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:41 dist-test-slave-q8mg krb5kdc[32116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:41 dist-test-slave-q8mg krb5kdc[32116](info): set up 2 sockets
Sep 25 02:35:41 dist-test-slave-q8mg krb5kdc[32116](info): commencing operation
krb5kdc: starting...
W20260925 02:35:42.763581 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:42 dist-test-slave-q8mg krb5kdc[32116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:42.886116 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34023
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:34023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:42.968436 32129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:42.968657 32129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:42.968690 32129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:42.969861 32129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:42.969901 32129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:42.969916 32129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:42.969929 32129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:42.969941 32129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:42.971423 32129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:34023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:42.971634 32129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:42.971813 32129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:42.976908 32137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:42.976925 32135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:42.977128 32129 server_base.cc:1034] running on GCE node
W20260925 02:35:42.976917 32134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:42.977479 32129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:42.977792 32129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:42.979014 32129 hybrid_clock.cc:648] HybridClock initialized: now 1790303742978998 us; error 25 us; skew 500 ppm
Sep 25 02:35:42 dist-test-slave-q8mg krb5kdc[32116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303742, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:42.988104 32129 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:42.988579 32129 webserver.cc:460] Webserver started at http://127.10.13.62:45389/ using document root <none> and password file <none>
I20260925 02:35:42.988845 32129 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:42.988971 32129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:42.989131 32129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:42.990221 32129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "aac6bf57069941818b97e754b87f9283"
format_stamp: "Formatted at 2026-09-25 02:35:42 on dist-test-slave-q8mg"
server_key: "f3ab0b3e4a8a1720134c18b743abb984"
server_key_iv: "62f02005b9c59dbe9eabc4419c2e4284"
server_key_version: "encryptionkey@0"
I20260925 02:35:42.990586 32129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "aac6bf57069941818b97e754b87f9283"
format_stamp: "Formatted at 2026-09-25 02:35:42 on dist-test-slave-q8mg"
server_key: "f3ab0b3e4a8a1720134c18b743abb984"
server_key_iv: "62f02005b9c59dbe9eabc4419c2e4284"
server_key_version: "encryptionkey@0"
I20260925 02:35:42.991969 32129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:42.992775 32144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:42.993022 32129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:42.993110 32129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "aac6bf57069941818b97e754b87f9283"
format_stamp: "Formatted at 2026-09-25 02:35:42 on dist-test-slave-q8mg"
server_key: "f3ab0b3e4a8a1720134c18b743abb984"
server_key_iv: "62f02005b9c59dbe9eabc4419c2e4284"
server_key_version: "encryptionkey@0"
I20260925 02:35:42.993187 32129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:42.996608 32129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:42.998495 32129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:42.998652 32129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:43.002696 32129 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:34023
I20260925 02:35:43.002730 32197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:34023 every 8 connection(s)
I20260925 02:35:43.003068 32129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:43.006067 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:43.009101 32198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283: Bootstrap starting.
I20260925 02:35:43.009795 32198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:43.010144 32198 log.cc:824] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:43.010995 32198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283: No bootstrap required, opened a new log
I20260925 02:35:43.011194 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 32129
I20260925 02:35:43.011327 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:43.011538 10292 external_mini_cluster.cc:1428] Setting key d981211460a03d0a3966329d698193ae
I20260925 02:35:43.013842 32198 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac6bf57069941818b97e754b87f9283" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34023 } }
I20260925 02:35:43.014020 32198 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:43.014060 32198 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac6bf57069941818b97e754b87f9283, State: Initialized, Role: FOLLOWER
I20260925 02:35:43.014207 32198 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac6bf57069941818b97e754b87f9283" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34023 } }
I20260925 02:35:43.014315 32198 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:43.014364 32198 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:43.014415 32198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:43.015115 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:43.012081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41445 (local address 127.10.13.62:34023)
0925 02:35:43.012284 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:43.012293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:43.012328 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:35:43.013786 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:43.013796 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:43.013836 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:43.013987 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:43.014736 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:43.014739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:43.014739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:43.014744 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:43.014781 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:43.014791 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:43.014848 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:43.014940 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 02:35:43.014973 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260925 02:35:43.015148 32198 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac6bf57069941818b97e754b87f9283" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34023 } }
I20260925 02:35:43.015259 32198 leader_election.cc:302] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac6bf57069941818b97e754b87f9283; no voters:
I20260925 02:35:43.015434 32198 leader_election.cc:288] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:43.015465 32203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:43.015582 32203 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [term 1 LEADER]: Becoming Leader. State: Replica: aac6bf57069941818b97e754b87f9283, State: Running, Role: LEADER
I20260925 02:35:43.015653 32203 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac6bf57069941818b97e754b87f9283" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34023 } }
I20260925 02:35:43.015688 32198 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:43.016026 32204 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac6bf57069941818b97e754b87f9283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac6bf57069941818b97e754b87f9283" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34023 } } }
I20260925 02:35:43.016099 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:43.016031 32205 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac6bf57069941818b97e754b87f9283. Latest consensus state: current_term: 1 leader_uuid: "aac6bf57069941818b97e754b87f9283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac6bf57069941818b97e754b87f9283" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 34023 } } }
I20260925 02:35:43.016145 32204 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:43.016155 32205 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283 [sys.catalog]: This master's current role is: LEADER
W20260925 02:35:43.017690 32219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:43.017756 32219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:43.018054 32209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:43.018322 32209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:43.019045 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:43.017778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57471 (local address 127.10.13.62:34023)
0925 02:35:43.017814 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:43.017816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:43.018072 (+ 256us) server_negotiation.cc:407] Connection header received
0925 02:35:43.018138 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:43.018139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:43.018144 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:43.018207 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:43.018814 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:43.018815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:43.018815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:43.018820 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:43.018834 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:43.018838 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:43.018899 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:43.018971 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:35:43.018983 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:35:43.020151 32209 catalog_manager.cc:1347] Generated new cluster ID: 617c6cdc573b49738b170f724fbb3f7c
I20260925 02:35:43.020200 32209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:43.031833 32209 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:43.032033 32209 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:43.035784 32209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac6bf57069941818b97e754b87f9283: Generated new TSK 0
I20260925 02:35:43.035970 32209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:43.039986 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 32129
2026-09-25T02:35:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:43 dist-test-slave-q8mg krb5kdc[32230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:43 dist-test-slave-q8mg krb5kdc[32230](info): set up 2 sockets
Sep 25 02:35:43 dist-test-slave-q8mg krb5kdc[32230](info): commencing operation
krb5kdc: starting...
W20260925 02:35:44.076251 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:44 dist-test-slave-q8mg krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:44.205282 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:33897
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:44.287588 32243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:44.287837 32243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:44.287874 32243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:44.289042 32243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:44.289110 32243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:44.289134 32243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:44.289157 32243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:44.289186 32243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:44.290668 32243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:44.290953 32243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:44.291188 32243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:44.296005 32248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:44.296020 32249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:44.296020 32251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:44.296335 32243 server_base.cc:1034] running on GCE node
I20260925 02:35:44.296576 32243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:44.296857 32243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:44.298054 32243 hybrid_clock.cc:648] HybridClock initialized: now 1790303744298041 us; error 29 us; skew 500 ppm
Sep 25 02:35:44 dist-test-slave-q8mg krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303744, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:44.300592 32243 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:44.301043 32243 webserver.cc:460] Webserver started at http://127.10.13.62:46705/ using document root <none> and password file <none>
I20260925 02:35:44.301249 32243 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:44.301291 32243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:44.301385 32243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:44.302361 32243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "28c404928f8d4c9fb3bb815d1135acfa"
format_stamp: "Formatted at 2026-09-25 02:35:44 on dist-test-slave-q8mg"
server_key: "0216e2e3c30fac222e1e897312b64b0f"
server_key_iv: "2341b605f3f9f0bb14847501e09eaf64"
server_key_version: "encryptionkey@0"
I20260925 02:35:44.302647 32243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "28c404928f8d4c9fb3bb815d1135acfa"
format_stamp: "Formatted at 2026-09-25 02:35:44 on dist-test-slave-q8mg"
server_key: "0216e2e3c30fac222e1e897312b64b0f"
server_key_iv: "2341b605f3f9f0bb14847501e09eaf64"
server_key_version: "encryptionkey@0"
I20260925 02:35:44.303915 32243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:35:44.304584 32258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:44.304773 32243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:35:44.304858 32243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "28c404928f8d4c9fb3bb815d1135acfa"
format_stamp: "Formatted at 2026-09-25 02:35:44 on dist-test-slave-q8mg"
server_key: "0216e2e3c30fac222e1e897312b64b0f"
server_key_iv: "2341b605f3f9f0bb14847501e09eaf64"
server_key_version: "encryptionkey@0"
I20260925 02:35:44.304962 32243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:44.322453 32243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:44.324986 32243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:44.325101 32243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:44.328802 32243 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:33897
I20260925 02:35:44.328868 32311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:33897 every 8 connection(s)
I20260925 02:35:44.329263 32243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:44.330230 32312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:44.330477 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 32243
I20260925 02:35:44.330595 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:44.330825 10292 external_mini_cluster.cc:1428] Setting key 283cc8c9e92586080434a359389c6125
I20260925 02:35:44.333855 32312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa: Bootstrap starting.
I20260925 02:35:44.334601 32312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:44.334930 32312 log.cc:824] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:44.335676 32312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa: No bootstrap required, opened a new log
I20260925 02:35:44.337921 32312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c404928f8d4c9fb3bb815d1135acfa" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33897 } }
Sep 25 02:35:44 dist-test-slave-q8mg krb5kdc[32230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:44.338100 32312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:44.338141 32312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c404928f8d4c9fb3bb815d1135acfa, State: Initialized, Role: FOLLOWER
I20260925 02:35:44.338286 32312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c404928f8d4c9fb3bb815d1135acfa" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33897 } }
I20260925 02:35:44.338380 32312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:44.338430 32312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:44.338485 32312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:44.339077 32312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c404928f8d4c9fb3bb815d1135acfa" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33897 } }
I20260925 02:35:44.339208 32312 leader_election.cc:302] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c404928f8d4c9fb3bb815d1135acfa; no voters:
I20260925 02:35:44.339412 32312 leader_election.cc:288] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:44.339442 32317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:44.339556 32317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [term 1 LEADER]: Becoming Leader. State: Replica: 28c404928f8d4c9fb3bb815d1135acfa, State: Running, Role: LEADER
I20260925 02:35:44.339643 32317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c404928f8d4c9fb3bb815d1135acfa" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33897 } }
I20260925 02:35:44.339711 32312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:44.340022 32317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c404928f8d4c9fb3bb815d1135acfa. Latest consensus state: current_term: 1 leader_uuid: "28c404928f8d4c9fb3bb815d1135acfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c404928f8d4c9fb3bb815d1135acfa" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33897 } } }
I20260925 02:35:44.340054 32319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28c404928f8d4c9fb3bb815d1135acfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c404928f8d4c9fb3bb815d1135acfa" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33897 } } }
I20260925 02:35:44.340167 32317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:44.340392 32319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:44.341075 32324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:44.341732 32324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:44.342618 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:44.331377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36727 (local address 127.10.13.62:33897)
0925 02:35:44.331573 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:44.331592 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:44.331619 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:35:44.332072 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:44.332078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:44.332128 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:44.332250 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:44.333139 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:44.334181 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:44.335295 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:44.335819 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:44.335939 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:44.338697 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:44.338701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:44.338702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:44.338715 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:44.340795 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:44.341271 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:44.341276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:44.341276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:44.341405 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:44.341671 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:44.341672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:44.341673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:44.341979 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:44.342146 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:44.342254 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 02:35:44.342423 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260925 02:35:44.343632 32324 catalog_manager.cc:1347] Generated new cluster ID: b92baa95bc27464a850da53960c55791
I20260925 02:35:44.343688 32324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:44.344942 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:44.359521 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:44.346282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50223 (local address 127.10.13.62:33897)
0925 02:35:44.346320 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:44.346322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:44.353110 (+ 6788us) server_negotiation.cc:407] Connection header received
0925 02:35:44.353196 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:44.353198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:44.353210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:44.353280 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:44.353892 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:44.354400 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:44.355128 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:44.355373 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:44.355452 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:44.356739 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:44.356741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:44.356741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:44.356749 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:44.358304 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:44.358689 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:44.358690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:44.358691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:44.358755 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:44.358978 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:44.358979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:44.358980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:44.359098 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:44.359203 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:44.359322 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 02:35:44.359404 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:35:44.361110 32324 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:44.361639 32324 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:44.368372 32324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28c404928f8d4c9fb3bb815d1135acfa: Generated new TSK 0
I20260925 02:35:44.368582 32324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:44.374120 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 32243
2026-09-25T02:35:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:44 dist-test-slave-q8mg krb5kdc[32343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:44 dist-test-slave-q8mg krb5kdc[32343](info): set up 2 sockets
Sep 25 02:35:44 dist-test-slave-q8mg krb5kdc[32343](info): commencing operation
krb5kdc: starting...
W20260925 02:35:45.415614 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:45 dist-test-slave-q8mg krb5kdc[32343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:45.539772 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:36533
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:45.622709 32356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:45.622931 32356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:45.622954 32356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:45.624181 32356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:45.624228 32356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:45.624243 32356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:45.624256 32356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:45.624298 32356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:45.625918 32356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:36533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:45.626197 32356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:45.626467 32356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:45.631537 32363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:45.631537 32365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:45.631593 32362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:45.631803 32356 server_base.cc:1034] running on GCE node
I20260925 02:35:45.632107 32356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:45.632413 32356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:45.633579 32356 hybrid_clock.cc:648] HybridClock initialized: now 1790303745633562 us; error 45 us; skew 500 ppm
Sep 25 02:35:45 dist-test-slave-q8mg krb5kdc[32343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303745, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:45.636119 32356 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:45.636670 32356 webserver.cc:460] Webserver started at http://127.10.13.62:40879/ using document root <none> and password file <none>
I20260925 02:35:45.636971 32356 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:45.637040 32356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:45.637202 32356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:45.638293 32356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "05b73530e5e34a3282d28a1f4f164242"
format_stamp: "Formatted at 2026-09-25 02:35:45 on dist-test-slave-q8mg"
server_key: "6f9cdb64002a07d5bc0d4373fedb3cdf"
server_key_iv: "83fb3be1217af235d3d6f0a3327644e3"
server_key_version: "encryptionkey@0"
I20260925 02:35:45.638675 32356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "05b73530e5e34a3282d28a1f4f164242"
format_stamp: "Formatted at 2026-09-25 02:35:45 on dist-test-slave-q8mg"
server_key: "6f9cdb64002a07d5bc0d4373fedb3cdf"
server_key_iv: "83fb3be1217af235d3d6f0a3327644e3"
server_key_version: "encryptionkey@0"
I20260925 02:35:45.640095 32356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:45.640995 32372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:45.641192 32356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:45.641295 32356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "05b73530e5e34a3282d28a1f4f164242"
format_stamp: "Formatted at 2026-09-25 02:35:45 on dist-test-slave-q8mg"
server_key: "6f9cdb64002a07d5bc0d4373fedb3cdf"
server_key_iv: "83fb3be1217af235d3d6f0a3327644e3"
server_key_version: "encryptionkey@0"
I20260925 02:35:45.641393 32356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:45.658227 32356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:45.660718 32356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:45.660918 32356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:45.665210 32356 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:36533
I20260925 02:35:45.665271 32424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:36533 every 8 connection(s)
I20260925 02:35:45.665786 32356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:45.666529 32425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:45.669130 32425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242: Bootstrap starting.
I20260925 02:35:45.669885 32425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:45.670213 32425 log.cc:824] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:45.670969 32425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242: No bootstrap required, opened a new log
I20260925 02:35:45.672722 32425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b73530e5e34a3282d28a1f4f164242" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 36533 } }
I20260925 02:35:45.672938 32425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:45.673002 32425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b73530e5e34a3282d28a1f4f164242, State: Initialized, Role: FOLLOWER
I20260925 02:35:45.673200 32425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b73530e5e34a3282d28a1f4f164242" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 36533 } }
I20260925 02:35:45.673280 32425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:45.673344 32425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:45.673404 32425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:45.674475 32425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b73530e5e34a3282d28a1f4f164242" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 36533 } }
I20260925 02:35:45.674615 32425 leader_election.cc:302] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05b73530e5e34a3282d28a1f4f164242; no voters:
I20260925 02:35:45.674810 32425 leader_election.cc:288] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:45.674830 32428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:45.675104 32425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:45.675215 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 32356
I20260925 02:35:45.675366 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:45.675570 10292 external_mini_cluster.cc:1428] Setting key 45b6f14e2a002dff96276959d4f116f5
I20260925 02:35:45.675712 32428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [term 1 LEADER]: Becoming Leader. State: Replica: 05b73530e5e34a3282d28a1f4f164242, State: Running, Role: LEADER
I20260925 02:35:45.675827 32428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b73530e5e34a3282d28a1f4f164242" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 36533 } }
I20260925 02:35:45.676474 32429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05b73530e5e34a3282d28a1f4f164242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b73530e5e34a3282d28a1f4f164242" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 36533 } } }
I20260925 02:35:45.676465 32430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05b73530e5e34a3282d28a1f4f164242. Latest consensus state: current_term: 1 leader_uuid: "05b73530e5e34a3282d28a1f4f164242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b73530e5e34a3282d28a1f4f164242" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 36533 } } }
I20260925 02:35:45.676617 32429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:45.676630 32430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:45.677029 32445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:45.677561 32445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:45.679051 32445 catalog_manager.cc:1347] Generated new cluster ID: f5eea85183504ad7ae7959b5870fa256
I20260925 02:35:45.679100 32445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:45.682928 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:45.676346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50371 (local address 127.10.13.62:36533)
0925 02:35:45.676905 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:45.676917 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:45.676959 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:35:45.677095 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:45.677103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:45.677167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:45.677347 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:45.677593 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:45.679122 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:45.680401 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:45.681174 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:45.681305 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:45.682081 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:45.682086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:45.682086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:45.682376 (+ 290us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:45.682420 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:45.682438 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:45.682518 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:45.682585 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:35:45.682657 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":72,"threads_started":1}
I20260925 02:35:45.683981 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:45.690085 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:45.685683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37089 (local address 127.10.13.62:36533)
0925 02:35:45.685721 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:45.685723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:45.685807 (+ 84us) server_negotiation.cc:407] Connection header received
0925 02:35:45.685915 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:45.685917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:45.685927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:45.685959 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:45.686180 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:45.687369 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:45.688536 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:45.688906 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:45.688959 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:45.689793 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:45.689795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:45.689795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:45.689801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:45.689817 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:45.689823 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:45.689855 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:45.689975 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 02:35:45.690004 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:35:45.705118 32445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:45.705948 32445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:45.709709 32445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05b73530e5e34a3282d28a1f4f164242: Generated new TSK 0
I20260925 02:35:45.709910 32445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:45.712801 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 32356
2026-09-25T02:35:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:45 dist-test-slave-q8mg krb5kdc[32456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:45 dist-test-slave-q8mg krb5kdc[32456](info): set up 2 sockets
Sep 25 02:35:45 dist-test-slave-q8mg krb5kdc[32456](info): commencing operation
krb5kdc: starting...
W20260925 02:35:46.751472 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:46 dist-test-slave-q8mg krb5kdc[32456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:46.871912 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:41549
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:46.959229 32469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:46.959482 32469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:46.959520 32469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:46.960686 32469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:46.960762 32469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:46.960788 32469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:46.960810 32469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:46.960839 32469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:46.962373 32469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:41549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:46.962651 32469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:46.962898 32469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:46.967949 32474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:46.967948 32475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:46.968053 32477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:46.968073 32469 server_base.cc:1034] running on GCE node
I20260925 02:35:46.968444 32469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:46.968767 32469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:46.969942 32469 hybrid_clock.cc:648] HybridClock initialized: now 1790303746969920 us; error 58 us; skew 500 ppm
Sep 25 02:35:46 dist-test-slave-q8mg krb5kdc[32456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303746, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:46.972379 32469 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:46.973004 32469 webserver.cc:460] Webserver started at http://127.10.13.62:41415/ using document root <none> and password file <none>
I20260925 02:35:46.973265 32469 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:46.973340 32469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:46.973475 32469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:46.974551 32469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "6b8e9161542545ba97e75682b6cfc0ee"
format_stamp: "Formatted at 2026-09-25 02:35:46 on dist-test-slave-q8mg"
server_key: "e02303a7cc8e91adc1b74916ceb0e6c2"
server_key_iv: "80815eaa098d14b28e14af5eced8118a"
server_key_version: "encryptionkey@0"
I20260925 02:35:46.974924 32469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "6b8e9161542545ba97e75682b6cfc0ee"
format_stamp: "Formatted at 2026-09-25 02:35:46 on dist-test-slave-q8mg"
server_key: "e02303a7cc8e91adc1b74916ceb0e6c2"
server_key_iv: "80815eaa098d14b28e14af5eced8118a"
server_key_version: "encryptionkey@0"
I20260925 02:35:46.976253 32469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:46.977052 32484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:46.977223 32469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:46.977329 32469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "6b8e9161542545ba97e75682b6cfc0ee"
format_stamp: "Formatted at 2026-09-25 02:35:46 on dist-test-slave-q8mg"
server_key: "e02303a7cc8e91adc1b74916ceb0e6c2"
server_key_iv: "80815eaa098d14b28e14af5eced8118a"
server_key_version: "encryptionkey@0"
I20260925 02:35:46.977413 32469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:46.980654 32469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:46.982735 32469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:46.982909 32469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:46.987133 32469 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:41549
I20260925 02:35:46.987165 32537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:41549 every 8 connection(s)
I20260925 02:35:46.987516 32469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:46.988446 32538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:46.991358 32538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee: Bootstrap starting.
I20260925 02:35:46.991962 32538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:46.992254 32538 log.cc:824] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:46.993041 32538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee: No bootstrap required, opened a new log
I20260925 02:35:46.994648 32538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8e9161542545ba97e75682b6cfc0ee" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41549 } }
I20260925 02:35:46.994796 32538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:35:46.994818 32538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8e9161542545ba97e75682b6cfc0ee, State: Initialized, Role: FOLLOWER
I20260925 02:35:46.995015 32538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8e9161542545ba97e75682b6cfc0ee" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41549 } }
I20260925 02:35:46.995098 32538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:35:46.995142 32538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:35:46.995193 32538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:46.996018 32538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8e9161542545ba97e75682b6cfc0ee" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41549 } }
I20260925 02:35:46.996109 32538 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b8e9161542545ba97e75682b6cfc0ee; no voters:
I20260925 02:35:46.996286 32538 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:35:46.996407 32541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:46.996583 32538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:46.996591 32541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [term 1 LEADER]: Becoming Leader. State: Replica: 6b8e9161542545ba97e75682b6cfc0ee, State: Running, Role: LEADER
I20260925 02:35:46.996729 32541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8e9161542545ba97e75682b6cfc0ee" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41549 } }
I20260925 02:35:46.997092 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 32469
I20260925 02:35:46.997219 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:46.997443 10292 external_mini_cluster.cc:1428] Setting key ca09298de6a4bb87eb9d633ce49acce8
I20260925 02:35:46.997880 32542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b8e9161542545ba97e75682b6cfc0ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8e9161542545ba97e75682b6cfc0ee" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41549 } } }
I20260925 02:35:46.998001 32542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:46.997900 32543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8e9161542545ba97e75682b6cfc0ee. Latest consensus state: current_term: 1 leader_uuid: "6b8e9161542545ba97e75682b6cfc0ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8e9161542545ba97e75682b6cfc0ee" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 41549 } } }
I20260925 02:35:46.998432 32543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee [sys.catalog]: This master's current role is: LEADER
W20260925 02:35:46.998649 32557 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:46.998713 32557 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:46.998771 32558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:46.998948 32558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:46.999938 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:46.998066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36713 (local address 127.10.13.62:41549)
0925 02:35:46.998422 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:46.998431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:46.998466 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:35:46.998588 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:46.998595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:46.998627 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:46.999051 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:46.999520 (+ 469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:46.999523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:46.999524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:46.999527 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:46.999567 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:46.999581 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:46.999642 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:46.999725 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 02:35:46.999765 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":57,"threads_started":1}
I20260925 02:35:47.000952 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:47.005540 32558 catalog_manager.cc:1347] Generated new cluster ID: ac86ea3e757249209a38a9b42251d66e
I20260925 02:35:47.005622 32558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:47.006897 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:47.002489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53399 (local address 127.10.13.62:41549)
0925 02:35:47.002529 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:47.002530 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:47.005738 (+ 3208us) server_negotiation.cc:407] Connection header received
0925 02:35:47.005765 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:47.005766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:47.005773 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:47.005813 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:47.006606 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:47.006607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:47.006608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:35:47.006612 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:47.006627 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:35:47.006631 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:47.006688 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:47.006784 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 02:35:47.006801 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:35:47.015283 32558 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:47.015437 32558 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:47.022455 32558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b8e9161542545ba97e75682b6cfc0ee: Generated new TSK 0
I20260925 02:35:47.022641 32558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:35:47.028193 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 32469
2026-09-25T02:35:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1315 ms)
[----------] 18 tests from AuthTokenIssuingTest (24108 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:47 dist-test-slave-q8mg krb5kdc[32569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:47 dist-test-slave-q8mg krb5kdc[32569](info): set up 2 sockets
Sep 25 02:35:47 dist-test-slave-q8mg krb5kdc[32569](info): commencing operation
krb5kdc: starting...
W20260925 02:35:48.063602 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:48.184831 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:33883
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:33883,127.10.13.61:44211,127.10.13.60:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:48.267674 32582 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:35:48.267904 32582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:48.267922 32582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:48.267943 32582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:48.269112 32582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:48.269153 32582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:48.269168 32582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:48.269181 32582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:48.269193 32582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:48.270812 32582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:33883,127.10.13.61:44211,127.10.13.60:37315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:33883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:48.271085 32582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:48.271323 32582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:48.276219 32588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:48.276226 32587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:48.276394 32582 server_base.cc:1034] running on GCE node
W20260925 02:35:48.276219 32590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:48.276853 32582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:48.277228 32582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:48.278407 32582 hybrid_clock.cc:648] HybridClock initialized: now 1790303748278374 us; error 48 us; skew 500 ppm
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:48.287704 32582 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:48.288321 32582 webserver.cc:460] Webserver started at http://127.10.13.62:35343/ using document root <none> and password file <none>
I20260925 02:35:48.288587 32582 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:48.288658 32582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:48.288820 32582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:48.290059 32582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "89167da8a0814ea781dad6e20083756b"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "8e52f5742c7c0ef3823b287175b9a47e"
server_key_iv: "0c3babf7f9ea76d79723bee2b6c07ee9"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.290503 32582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "89167da8a0814ea781dad6e20083756b"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "8e52f5742c7c0ef3823b287175b9a47e"
server_key_iv: "0c3babf7f9ea76d79723bee2b6c07ee9"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.291939 32582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:48.292801 32597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:48.293046 32582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:35:48.293167 32582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "89167da8a0814ea781dad6e20083756b"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "8e52f5742c7c0ef3823b287175b9a47e"
server_key_iv: "0c3babf7f9ea76d79723bee2b6c07ee9"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.293246 32582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:48.304361 32582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:48.306664 32582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:48.306831 32582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:48.311106 32582 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:33883
I20260925 02:35:48.311156 32649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:33883 every 8 connection(s)
I20260925 02:35:48.311524 32582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:48.312498 32650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:48.314134 32650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:48.320251 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 32582
I20260925 02:35:48.320387 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:48.320613 10292 external_mini_cluster.cc:1428] Setting key a478df5e065624d9a811025b5f938e54
I20260925 02:35:48.324174 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.314842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.10.13.62:33883)
0925 02:35:48.315584 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.315588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.315598 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:35:48.315657 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.315658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.315672 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.315733 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.316259 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.317178 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.317993 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.318291 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.318371 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.320360 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.320371 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.320371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.320380 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.322401 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.322963 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.322966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.322967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.323057 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.323338 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.323339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.323339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.323559 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.323629 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.323879 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.324002 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":715,"thread_start_us":78,"threads_started":1}
I20260925 02:35:48.324282 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.314776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:33883 (local address 127.0.0.1:52978)
0925 02:35:48.314995 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.315032 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.315450 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.315780 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.315782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.316014 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.316169 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.316173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.317262 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.317263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.317921 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.317923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.318380 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.318381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.318434 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.318967 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.318976 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.320263 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.322503 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.322511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.322514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.322854 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.323140 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.323141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.323141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.323258 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.323634 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.323640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.323717 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.323827 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.324088 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":105,"thread_start_us":111,"threads_started":1}
I20260925 02:35:48.325080 32650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:35:48.325461 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.325327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47412)
0925 02:35:48.325381 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.325435 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 02:35:48.325634 32601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
I20260925 02:35:48.326253 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.326132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47414)
0925 02:35:48.326186 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.326220 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 02:35:48.326408 32650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:44211: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.10.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab.
Entry for principal kudu/127.10.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab.
I20260925 02:35:48.348701 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.61:44211
--webserver_interface=127.10.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab
--principal=kudu/127.10.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:33883,127.10.13.61:44211,127.10.13.60:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:35:48.375679 32650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } attempt: 1
I20260925 02:35:48.376335 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.376136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47430)
0925 02:35:48.376240 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.376294 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:35:48.376726 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.376634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47444)
0925 02:35:48.376684 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.376709 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 02:35:48.376849 32650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:44211: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
W20260925 02:35:48.439061 32658 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:35:48.439309 32658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:48.439329 32658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:48.439353 32658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:48.440510 32658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:48.440543 32658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:48.440557 32658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:48.440569 32658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:48.440582 32658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:48.442028 32658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:33883,127.10.13.61:44211,127.10.13.60:37315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab
--principal=kudu/127.10.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.61:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:48.442243 32658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:48.442430 32658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:35:48.445041 32650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } attempt: 2
I20260925 02:35:48.445480 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.445352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47448)
0925 02:35:48.445412 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.445453 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 02:35:48.445842 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.445739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47456)
0925 02:35:48.445787 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.445815 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 02:35:48.445938 32650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:44211: Network error: Client connection negotiation failed: client connection to 127.10.13.61:44211: connect: Connection refused (error 111)
W20260925 02:35:48.447887 32663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:48.447881 32664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:48.447883 32666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:48.447933 32658 server_base.cc:1034] running on GCE node
I20260925 02:35:48.448349 32658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:48.448689 32658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:48.449879 32658 hybrid_clock.cc:648] HybridClock initialized: now 1790303748449846 us; error 42 us; skew 500 ppm
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:48.452489 32658 init.cc:377] Logged in from keytab as kudu/127.10.13.61@KRBTEST.COM (short username kudu)
I20260925 02:35:48.453112 32658 webserver.cc:460] Webserver started at http://127.10.13.61:35925/ using document root <none> and password file <none>
I20260925 02:35:48.453354 32658 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:48.453423 32658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:48.453583 32658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:48.454707 32658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/data/instance:
uuid: "34c543c67b584f1ea0699d858b028dd2"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "1283aefb74075dabb896d5b119d8e067"
server_key_iv: "d0476cab6135358dda1dc9d8cf540d90"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.455083 32658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/wal/instance:
uuid: "34c543c67b584f1ea0699d858b028dd2"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "1283aefb74075dabb896d5b119d8e067"
server_key_iv: "d0476cab6135358dda1dc9d8cf540d90"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.456491 32658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:48.457382 32673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:48.457552 32658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:48.457664 32658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/wal
uuid: "34c543c67b584f1ea0699d858b028dd2"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "1283aefb74075dabb896d5b119d8e067"
server_key_iv: "d0476cab6135358dda1dc9d8cf540d90"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.457784 32658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:48.471422 32658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:48.474141 32658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:48.474319 32658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:48.478698 32658 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.61:44211
I20260925 02:35:48.478761 32725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.61:44211 every 8 connection(s)
I20260925 02:35:48.479146 32658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
I20260925 02:35:48.480203 32726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:48.481884 32726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:35:48.483909 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 32658
I20260925 02:35:48.483990 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-1/wal/instance
I20260925 02:35:48.484195 10292 external_mini_cluster.cc:1428] Setting key 38a984d15e2d778192bcff9b33f2ca4d
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.13.60@KRBTEST.COM; defaulting to no policy
I20260925 02:35:48.491694 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.482625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.10.13.62:33883)
0925 02:35:48.482761 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.482764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.482954 (+ 190us) server_negotiation.cc:407] Connection header received
0925 02:35:48.483537 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.483538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.483551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.483611 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.484330 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.484837 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.485909 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.486127 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.486210 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.488646 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.488648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.488648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.488657 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.490172 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.490598 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.490599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.490600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.490662 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.490932 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.490933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.490934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.491055 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.491126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.491463 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.491559 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 02:35:48.491752 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.482579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:33883 (local address 127.0.0.1:52994)
0925 02:35:48.482808 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.482855 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.483422 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.483688 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.483692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.483976 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.484242 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.484249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.485007 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.485008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.485798 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.485800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.486256 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.486258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.486379 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.486939 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.486948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.488544 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.490277 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.490285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.490288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.490519 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.490742 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.490743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.490743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.490858 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.491129 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.491137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.491285 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.491407 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.491557 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260925 02:35:48.492426 32726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.60@KRBTEST.COM" created.
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:35:48.501200 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.492806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.10.13.61:44211)
0925 02:35:48.492975 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.492980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.492999 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:35:48.493029 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.493031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.493042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.493103 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.493354 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.494396 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.495180 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.495665 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.495732 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.497747 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.497755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.497755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.497763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.499761 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.500221 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.500223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.500224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.500275 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.500497 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.500497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.500498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.500633 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.500694 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.500931 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.501055 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
I20260925 02:35:48.501225 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.492713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47470)
0925 02:35:48.492808 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.492819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.492866 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.493117 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.493118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.493231 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.493282 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.493283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.494523 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.494524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.495086 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.495088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.495793 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.495794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.495864 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.496384 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.496391 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.497661 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.499860 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.499862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.499862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.500131 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.500350 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.500351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.500352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.500428 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.500703 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.500704 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.500758 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.500844 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.501008 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 02:35:48.501976 32726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:35:48.502353 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.502209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51796)
0925 02:35:48.502263 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.502321 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 02:35:48.502472 32677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
I20260925 02:35:48.502898 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.502798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51802)
0925 02:35:48.502853 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.502879 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:35:48.503026 32726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:37315: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab.
Entry for principal kudu/127.10.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab.
I20260925 02:35:48.512233 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.60:37315
--webserver_interface=127.10.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab
--principal=kudu/127.10.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:33883,127.10.13.61:44211,127.10.13.60:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:35:48.537086 32650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } attempt: 3
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:35:48.544461 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.537432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47480)
0925 02:35:48.537503 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.537521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.537594 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.537763 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.537764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.537935 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.538000 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.538002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.538679 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.538681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.539199 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.539201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.539603 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.539604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.539691 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.540188 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.540194 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.541598 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.543224 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.543225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.543226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.543474 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.543725 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.543726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.543727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.543796 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.544113 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.544114 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.544168 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.544253 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.544358 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:35:48.544899 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.537515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.10.13.61:44211)
0925 02:35:48.537586 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.537588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.537600 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:35:48.537666 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.537668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.537682 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.537756 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.538090 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.538587 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.539292 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.539528 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.539605 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.541691 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.541692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.541693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.541703 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.543110 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.543562 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.543563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.543564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.543635 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.543875 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.543876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.543876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.544022 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.544095 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.544325 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.544461 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:35:48.546180 32650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:35:48.546555 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.546452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51806)
0925 02:35:48.546500 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.546526 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:35:48.546995 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.546889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51814)
0925 02:35:48.546940 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.546970 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:35:48.547133 32650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:37315: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
I20260925 02:35:48.552249 32726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } attempt: 1
I20260925 02:35:48.552618 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.552495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51818)
0925 02:35:48.552549 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.552589 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 02:35:48.552945 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.552837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51822)
0925 02:35:48.552899 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.552923 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:35:48.553045 32726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:37315: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
I20260925 02:35:48.578299 32650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } attempt: 1
I20260925 02:35:48.578860 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.578681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51834)
0925 02:35:48.578760 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.578815 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:35:48.579289 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.579180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51850)
0925 02:35:48.579234 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.579263 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:35:48.579427 32650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:37315: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
W20260925 02:35:48.598276 32733 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:35:48.598531 32733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:48.598572 32733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:48.598637 32733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:48.599792 32733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:48.599858 32733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:48.599884 32733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:48.599915 32733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:48.599941 32733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:48.601573 32733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:33883,127.10.13.61:44211,127.10.13.60:37315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab
--principal=kudu/127.10.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.60:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:48.601872 32733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:48.602101 32733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:48.607142 32741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:48.607155 32739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:48.607187 32733 server_base.cc:1034] running on GCE node
W20260925 02:35:48.607141 32738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:48.607650 32733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:48.607981 32733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:48.609163 32733 hybrid_clock.cc:648] HybridClock initialized: now 1790303748609140 us; error 40 us; skew 500 ppm
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:48.611833 32733 init.cc:377] Logged in from keytab as kudu/127.10.13.60@KRBTEST.COM (short username kudu)
I20260925 02:35:48.612300 32733 webserver.cc:460] Webserver started at http://127.10.13.60:41203/ using document root <none> and password file <none>
I20260925 02:35:48.612507 32733 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:48.612548 32733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:48.612643 32733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:48.613757 32733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/data/instance:
uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "62d6afe69fbea8873ed9ffad739b64af"
server_key_iv: "a6dbd6e194c375863923f69c3f8cd9d1"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.614064 32733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/wal/instance:
uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "62d6afe69fbea8873ed9ffad739b64af"
server_key_iv: "a6dbd6e194c375863923f69c3f8cd9d1"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.615311 32733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:35:48.616041 32748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:48.616218 32733 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:35:48.616310 32733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/wal
uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc"
format_stamp: "Formatted at 2026-09-25 02:35:48 on dist-test-slave-q8mg"
server_key: "62d6afe69fbea8873ed9ffad739b64af"
server_key_iv: "a6dbd6e194c375863923f69c3f8cd9d1"
server_key_version: "encryptionkey@0"
I20260925 02:35:48.616365 32733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:48.621183 32726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } attempt: 2
I20260925 02:35:48.621625 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.621491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51852)
0925 02:35:48.621554 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.621594 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:35:48.622002 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.621922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51862)
0925 02:35:48.621962 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.621985 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 02:35:48.622133 32726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:37315: Network error: Client connection negotiation failed: client connection to 127.10.13.60:37315: connect: Connection refused (error 111)
I20260925 02:35:48.631793 32733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:48.633844 32733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:48.633958 32733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:48.638006 32733 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.60:37315
I20260925 02:35:48.638075 332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.60:37315 every 8 connection(s)
I20260925 02:35:48.638422 32733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
I20260925 02:35:48.639496 333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:48.641121 333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:48.647243 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 32733
I20260925 02:35:48.647353 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790303478591324-10292-0/minicluster-data/master-2/wal/instance
I20260925 02:35:48.647578 10292 external_mini_cluster.cc:1428] Setting key 48fc85ccb59482ad14f3d58759b14e85
I20260925 02:35:48.650918 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.641860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:33883 (local address 127.0.0.1:52996)
0925 02:35:48.642063 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.642107 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.642560 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.642866 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.642869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.643112 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.643308 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.643313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.644123 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.644124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.644977 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.644979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.645346 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.645347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.645449 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.645987 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.645995 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.647534 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.649527 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.649532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.649535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.649829 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.650040 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.650041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.650041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.650175 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.650437 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.650441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.650554 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.650636 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.650751 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":122,"threads_started":1}
I20260925 02:35:48.650924 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.641905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.10.13.62:33883)
0925 02:35:48.641974 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.641976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.642218 (+ 242us) server_negotiation.cc:407] Connection header received
0925 02:35:48.642674 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.642675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.642686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.642760 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.643408 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.644027 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.645065 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.645256 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.645309 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.647645 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.647647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.647647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.647656 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.649456 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.649916 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.649917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.649917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.649972 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.650239 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.650239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.650240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.650340 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.650428 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.650737 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.650822 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:35:48.651502 333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
I20260925 02:35:48.654560 32650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } attempt: 2
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:35:48.658026 337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.648044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.10.13.60:37315)
0925 02:35:48.648260 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.648272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.648288 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:35:48.648362 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.648365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.648377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.648440 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.649291 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.650307 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.651390 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.651732 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.651815 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.654777 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.654789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.654789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.654797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.656671 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.657074 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.657078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.657079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.657142 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.657388 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.657389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.657390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.657573 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.657720 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.657820 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.657900 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260925 02:35:48.659358 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.651844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.10.13.61:44211)
0925 02:35:48.651886 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.651888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.651899 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:35:48.651926 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.651927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.651937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.651995 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.652256 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.652782 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.653652 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.654203 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.654317 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.656347 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.656349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.656349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.656356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.658063 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.658504 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.658506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.658506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.658566 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.658760 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.658761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.658762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.658899 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.658973 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.659170 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.659257 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
I20260925 02:35:48.659649 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.651785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:44211 (local address 127.0.0.1:47490)
0925 02:35:48.651825 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.651835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.651870 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.651998 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.651999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.652128 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.652168 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.652170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.652890 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.652891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.653560 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.653563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.654324 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.654326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.654401 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.654963 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.654969 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.656259 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.658249 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.658250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.658250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.658425 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.658621 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.658622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.658622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.658684 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.658981 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.658982 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.659028 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.659442 (+ 414us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.659538 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 02:35:48.659997 333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:35:48.663616 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.654934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.10.13.60:37315)
0925 02:35:48.655317 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.655319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.655327 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:35:48.655347 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.655348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.655360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.655738 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.655799 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.656532 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.657565 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.657878 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.658263 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.660003 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.660005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.660005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.660011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.662419 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.662766 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.662767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.662767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.662824 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.663081 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.663083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.663083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.663221 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.663288 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.663470 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.663540 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":90,"threads_started":1}
I20260925 02:35:48.663621 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.654826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51894)
0925 02:35:48.654889 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.654900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.654944 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.655471 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.655472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.655627 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.655667 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.655669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.656649 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.656651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.657489 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.657492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.657970 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.657971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.658024 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.658417 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.658422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.659913 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.662505 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.662506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.662507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.662675 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.662918 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.662920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.662920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.662997 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.663296 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.663297 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.663341 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.663431 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.663525 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
I20260925 02:35:48.665794 32650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b: Bootstrap starting.
I20260925 02:35:48.666603 32650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:48.666953 32650 log.cc:824] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:48.667506 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.659112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.10.13.62:33883)
0925 02:35:48.659148 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.659149 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.659156 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:35:48.659174 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.659175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.659183 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.659237 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.659819 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.660500 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.661574 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.661843 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.661885 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.664509 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.664510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.664511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.664518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.666173 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.666586 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.666587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.666587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.666642 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.666866 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.666867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.666868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.666992 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.667095 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.667344 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.667416 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:35:48.667969 32650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b: No bootstrap required, opened a new log
I20260925 02:35:48.668084 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.660281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51896)
0925 02:35:48.660315 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.660325 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.660361 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.660753 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.660754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.660924 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.660952 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.660954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.661658 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.661659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.662351 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.662353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.662839 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.662840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.662911 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.664025 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.664029 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.665278 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.667011 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.667012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.667013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.667191 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.667397 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.667398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.667398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.667450 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.667766 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.667767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.667805 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.667878 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.667985 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 02:35:48.668397 337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.660598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.10.13.60:37315)
0925 02:35:48.660628 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.660630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.660636 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:35:48.660654 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.660654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.660662 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.660748 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.661033 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.661580 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.662433 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.662758 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.662840 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.665551 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.665552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.665552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.665558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.666912 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.667279 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.667280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.667280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.667332 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.667516 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.667517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.667517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.667680 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.667743 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.667938 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.668108 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":15}
I20260925 02:35:48.669870 32650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } }
I20260925 02:35:48.670017 333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc: Bootstrap starting.
I20260925 02:35:48.670019 32650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89167da8a0814ea781dad6e20083756b, State: Initialized, Role: FOLLOWER
I20260925 02:35:48.670109 32650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } }
I20260925 02:35:48.670449 32650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:48.670413 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.670545 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:48.670670 333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:48.670902 333 log.cc:824] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:48.671715 333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc: No bootstrap required, opened a new log
W20260925 02:35:48.672338 352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:48.672396 352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:35:48.674104 333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } }
I20260925 02:35:48.674283 333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9f784b6d8e4e2499bff1fb136ae6dc, State: Initialized, Role: FOLLOWER
I20260925 02:35:48.674371 333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } }
I20260925 02:35:48.674678 333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:48.675069 354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.675161 354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:48.673202 32582 master_runner.cc:186] Error getting master registration for 127.10.13.61:44211: OK, instance_id { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" instance_seqno: 1790303748477484 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:35:48.675668 32582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:35:48.677235 365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:48.677299 365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:48.678529 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.668098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.10.13.61:44211)
0925 02:35:48.668135 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.668137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.668205 (+ 68us) server_negotiation.cc:407] Connection header received
0925 02:35:48.668291 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.668292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.668303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.668351 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.668973 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.669652 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.671092 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.671425 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.671478 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.674560 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.674561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.674562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.674570 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.677018 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.677507 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.677508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.677509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.677607 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.677835 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.677836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.677836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.677984 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.678133 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.678352 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.678436 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:35:48.680102 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:35:48.681159 32624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "89167da8a0814ea781dad6e20083756b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53012
I20260925 02:35:48.681272 32624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:35:48.681316 32624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:48.679015 32733 master_runner.cc:186] Error getting master registration for 127.10.13.61:44211: OK, instance_id { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" instance_seqno: 1790303748477484 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:35:48.681365 32733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:35:48.682022 32624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } }
I20260925 02:35:48.682184 32624 leader_election.cc:288] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [CANDIDATE]: Term 1 election: Requested vote from peers 34c543c67b584f1ea0699d858b028dd2 (127.10.13.61:44211), 1d9f784b6d8e4e2499bff1fb136ae6dc (127.10.13.60:37315)
I20260925 02:35:48.682586 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89167da8a0814ea781dad6e20083756b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc"
I20260925 02:35:48.682693 307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:48.683599 307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89167da8a0814ea781dad6e20083756b in term 1.
I20260925 02:35:48.683800 32601 leader_election.cc:302] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9f784b6d8e4e2499bff1fb136ae6dc, 89167da8a0814ea781dad6e20083756b; no voters:
I20260925 02:35:48.683919 340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:35:48.684062 340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [term 1 LEADER]: Becoming Leader. State: Replica: 89167da8a0814ea781dad6e20083756b, State: Running, Role: LEADER
I20260925 02:35:48.684156 340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } }
I20260925 02:35:48.684615 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89167da8a0814ea781dad6e20083756b. Latest consensus state: current_term: 1 leader_uuid: "89167da8a0814ea781dad6e20083756b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.684716 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:48.688992 375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:48.689240 375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:48.691133 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.683357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.10.13.62:33883)
0925 02:35:48.683405 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.683407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.683510 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:35:48.683613 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.683614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.683626 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.683668 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.684506 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.685260 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.686256 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.686560 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.686625 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.688123 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.688124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.688125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.688133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.689835 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.690245 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.690246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.690246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.690307 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.690522 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.690523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.690523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.690650 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.690724 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.690951 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.691034 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:35:48.691242 307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [term 1 FOLLOWER]: Refusing update from remote peer 89167da8a0814ea781dad6e20083756b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:48.696931 372 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:48.699384 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.683994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.10.13.61:44211)
0925 02:35:48.684081 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.684083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.691607 (+ 7524us) server_negotiation.cc:407] Connection header received
0925 02:35:48.691716 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.691717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.691729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.691758 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.692421 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.692961 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.694353 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.694684 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.694751 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.696330 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.696331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.696331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.696339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.698025 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.698468 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.698469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.698470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.698549 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.698774 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.698775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.698775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.698922 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.699006 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.699183 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.699269 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:35:48.699446 354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89167da8a0814ea781dad6e20083756b. Latest consensus state: current_term: 1 leader_uuid: "89167da8a0814ea781dad6e20083756b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.699793 354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:48.706312 337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.684645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.10.13.60:37315)
0925 02:35:48.684900 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.684902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.697584 (+ 12682us) server_negotiation.cc:407] Connection header received
0925 02:35:48.697607 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.697608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.697618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.697681 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.699780 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.700508 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.701552 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.701861 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.701929 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.703424 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.703425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.703425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.703432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.705072 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.705473 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.705474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.705475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.705547 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.705752 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.705753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.705753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.705866 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.705989 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.706112 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.706190 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260925 02:35:48.710084 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89167da8a0814ea781dad6e20083756b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.710191 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:48.710193 354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89167da8a0814ea781dad6e20083756b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.710263 354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:48.710434 377 mvcc.cc:205] Tried to move back new op lower bound from 7333084154636046336 to 7333084154611081216. Current Snapshot: MvccSnapshot[applied={T|T < 7333084154636046336}]
I20260925 02:35:48.710558 378 mvcc.cc:205] Tried to move back new op lower bound from 7333084154636046336 to 7333084154611081216. Current Snapshot: MvccSnapshot[applied={T|T < 7333084154636046336}]
I20260925 02:35:48.713250 32726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } attempt: 3
Sep 25 02:35:48 dist-test-slave-q8mg krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303748, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
I20260925 02:35:48.717787 375 catalog_manager.cc:1347] Generated new cluster ID: dcfe486345954fc2890993f727193e52
I20260925 02:35:48.717844 375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:48.720841 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.713513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:37315 (local address 127.0.0.1:51924)
0925 02:35:48.713560 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:48.713573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:48.713621 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:48.713803 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:48.713804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:48.713946 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:48.713985 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.713987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.714885 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.714886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.715379 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:48.715380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.715832 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.715833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:48.715886 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.716336 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:48.716342 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:48.717655 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:48.719687 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.719689 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.719690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.719919 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.720124 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:48.720125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:48.720125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:48.720184 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:48.720493 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:48.720494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:48.720556 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 02:35:48.720623 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:35:48.720739 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 02:35:48.721086 337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:48.713565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.10.13.60:37315)
0925 02:35:48.713602 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.713603 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.713630 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:35:48.713704 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.713705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.713716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.713768 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.714063 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.714780 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.715477 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:48.715772 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:48.715834 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:48.717762 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:48.717764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:48.717764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:48.717772 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:48.719587 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.720013 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.720014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.720014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.720066 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:48.720259 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:48.720260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:48.720260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:48.720390 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:48.720478 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:48.720675 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 02:35:48.720799 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:35:48.722746 32726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2: Bootstrap starting.
I20260925 02:35:48.723479 32726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:48.723795 32726 log.cc:824] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:48.724772 32726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2: No bootstrap required, opened a new log
I20260925 02:35:48.727028 32726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } }
I20260925 02:35:48.727207 32726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c543c67b584f1ea0699d858b028dd2, State: Initialized, Role: FOLLOWER
I20260925 02:35:48.727345 32726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } }
I20260925 02:35:48.727587 375 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:48.727785 32726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:48.727878 32699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:48.728296 375 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:48.728928 32699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [term 1 FOLLOWER]: Refusing update from remote peer 89167da8a0814ea781dad6e20083756b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:48.729255 372 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:48.727769 383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.727842 32700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89167da8a0814ea781dad6e20083756b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "34c543c67b584f1ea0699d858b028dd2"
I20260925 02:35:48.730690 32700 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89167da8a0814ea781dad6e20083756b in term 1.
I20260925 02:35:48.731659 383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:48.733256 383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89167da8a0814ea781dad6e20083756b. Latest consensus state: current_term: 1 leader_uuid: "89167da8a0814ea781dad6e20083756b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.733368 383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:48.733433 383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89167da8a0814ea781dad6e20083756b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.733506 383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:48.734047 395 mvcc.cc:205] Tried to move back new op lower bound from 7333084154782027776 to 7333084154611081216. Current Snapshot: MvccSnapshot[applied={T|T < 7333084154636046336}]
I20260925 02:35:48.737604 394 catalog_manager.cc:1259] Loaded cluster ID: dcfe486345954fc2890993f727193e52
I20260925 02:35:48.737658 394 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2: loading cluster ID for follower catalog manager: success
I20260925 02:35:48.738822 394 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2: acquiring CA information for follower catalog manager: success
W20260925 02:35:48.738963 394 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260925 02:35:48.741611 372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89167da8a0814ea781dad6e20083756b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89167da8a0814ea781dad6e20083756b" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 33883 } } peers { permanent_uuid: "34c543c67b584f1ea0699d858b028dd2" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 44211 } } peers { permanent_uuid: "1d9f784b6d8e4e2499bff1fb136ae6dc" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 37315 } } }
I20260925 02:35:48.741715 372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:48.744984 375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89167da8a0814ea781dad6e20083756b: Generated new TSK 0
I20260925 02:35:48.745226 375 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:35:48.770677 32654 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.768959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.10.13.62:33883)
0925 02:35:48.769008 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.769010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.769960 (+ 950us) server_negotiation.cc:407] Connection header received
0925 02:35:48.770068 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.770069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.770081 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.770112 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.770581 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53028: BlockingRecv error: recv got EOF from 127.0.0.1:53028 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 02:35:48.771636 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.769231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.10.13.61:44211)
0925 02:35:48.769269 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.769271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.771109 (+ 1838us) server_negotiation.cc:407] Connection header received
0925 02:35:48.771209 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.771210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.771221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.771250 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.771572 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47516: BlockingRecv error: recv got EOF from 127.0.0.1:47516 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 02:35:48.772542 337 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.769683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.10.13.60:37315)
0925 02:35:48.769745 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.769747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.772000 (+ 2253us) server_negotiation.cc:407] Connection header received
0925 02:35:48.772131 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.772132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.772143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.772171 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.772480 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51930: BlockingRecv error: recv got EOF from 127.0.0.1:51930 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 02:35:48.775032 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.774236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.10.13.61:44211)
0925 02:35:48.774272 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.774273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.774500 (+ 227us) server_negotiation.cc:407] Connection header received
0925 02:35:48.774605 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.774606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.774624 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.774652 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.774987 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47522: BlockingRecv error: recv got EOF from 127.0.0.1:47522 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 02:35:48.778095 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.777342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.10.13.61:44211)
0925 02:35:48.777373 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.777375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.777609 (+ 234us) server_negotiation.cc:407] Connection header received
0925 02:35:48.777704 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.777705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.777729 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.777757 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.778051 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47536: BlockingRecv error: recv got EOF from 127.0.0.1:47536 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 02:35:48.781991 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.781223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.10.13.61:44211)
0925 02:35:48.781255 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.781257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.781512 (+ 255us) server_negotiation.cc:407] Connection header received
0925 02:35:48.781604 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.781605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.781615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.781643 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.781946 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47544: BlockingRecv error: recv got EOF from 127.0.0.1:47544 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 02:35:48.787832 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.787062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.10.13.61:44211)
0925 02:35:48.787096 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.787098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.787332 (+ 234us) server_negotiation.cc:407] Connection header received
0925 02:35:48.787427 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.787428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.787438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.787466 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.787786 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47560: BlockingRecv error: recv got EOF from 127.0.0.1:47560 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 02:35:48.798056 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.797138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.10.13.61:44211)
0925 02:35:48.797178 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.797180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.797471 (+ 291us) server_negotiation.cc:407] Connection header received
0925 02:35:48.797575 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.797576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.797588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.797617 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.797988 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47574: BlockingRecv error: recv got EOF from 127.0.0.1:47574 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 02:35:48.816444 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.815454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.10.13.61:44211)
0925 02:35:48.815500 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.815502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.815820 (+ 318us) server_negotiation.cc:407] Connection header received
0925 02:35:48.815936 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.815938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.815950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.815983 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.816387 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47584: BlockingRecv error: recv got EOF from 127.0.0.1:47584 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 02:35:48.850862 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.849876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.10.13.61:44211)
0925 02:35:48.849922 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.849924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.850234 (+ 310us) server_negotiation.cc:407] Connection header received
0925 02:35:48.850343 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.850344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.850356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.850387 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.850789 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47590: BlockingRecv error: recv got EOF from 127.0.0.1:47590 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 02:35:48.917273 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:48.916497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.10.13.61:44211)
0925 02:35:48.916597 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:48.916599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:48.916672 (+ 73us) server_negotiation.cc:407] Connection header received
0925 02:35:48.916787 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:48.916788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:48.916800 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:48.916931 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:48.917223 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47594: BlockingRecv error: recv got EOF from 127.0.0.1:47594 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 02:35:49.047444 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:49.046629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.10.13.61:44211)
0925 02:35:49.046723 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:49.046725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:49.046821 (+ 96us) server_negotiation.cc:407] Connection header received
0925 02:35:49.046926 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:49.046928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:49.046941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:49.047015 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:49.047372 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47602: BlockingRecv error: recv got EOF from 127.0.0.1:47602 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 02:35:49.305451 32730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:49.304733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.10.13.61:44211)
0925 02:35:49.304831 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:49.304834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:49.304888 (+ 54us) server_negotiation.cc:407] Connection header received
0925 02:35:49.304919 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:49.304920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:49.304933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:49.305012 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:49.305381 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47614: BlockingRecv error: recv got EOF from 127.0.0.1:47614 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:35:49.677748 365 catalog_manager.cc:1259] Loaded cluster ID: dcfe486345954fc2890993f727193e52
I20260925 02:35:49.677824 365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc: loading cluster ID for follower catalog manager: success
I20260925 02:35:49.678737 365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc: acquiring CA information for follower catalog manager: success
I20260925 02:35:49.679021 365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d9f784b6d8e4e2499bff1fb136ae6dc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:35:49.739601 394 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34c543c67b584f1ea0699d858b028dd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:35:49.821455 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:49.818656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.10.13.61:44211)
0925 02:35:49.818801 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:49.818803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:49.818833 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:35:49.818944 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:49.818945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:49.819010 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:49.819086 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:49.819249 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.819762 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.820574 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.820767 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.820825 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:49.821002 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:49.821117 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:49.821180 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:49.821306 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 02:35:49.821331 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260925 02:35:49.826146 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:49.823465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.10.13.60:37315)
0925 02:35:49.823566 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:49.823568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:49.823578 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:35:49.823699 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:49.823700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:49.823753 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:49.823850 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:49.823973 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.824476 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.825304 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.825533 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.825612 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:49.825787 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:49.825906 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:49.825989 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:49.826030 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 02:35:49.826052 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":48,"threads_started":1}
I20260925 02:35:49.830931 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:49.827969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.10.13.61:44211)
0925 02:35:49.828016 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:49.828017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:49.828169 (+ 152us) server_negotiation.cc:407] Connection header received
0925 02:35:49.828263 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:49.828265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:49.828276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:49.828327 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:49.828526 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.829316 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.830222 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.830431 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.830507 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:49.830665 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:49.830700 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:49.830745 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:49.830812 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:35:49.830835 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:35:49.831455 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:49.828275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.10.13.60:37315)
0925 02:35:49.828357 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:49.828359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:49.828573 (+ 214us) server_negotiation.cc:407] Connection header received
0925 02:35:49.828638 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:49.828638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:49.828650 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:49.828698 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:49.828832 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.829338 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.830608 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.830929 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.830993 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:49.831135 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:49.831166 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:49.831191 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:49.831337 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 02:35:49.831359 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:35:49.831728 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:49.827953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.10.13.62:33883)
0925 02:35:49.828063 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:49.828065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:49.828139 (+ 74us) server_negotiation.cc:407] Connection header received
0925 02:35:49.828253 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:49.828254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:49.828310 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:49.828439 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:49.828607 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.829382 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.830577 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:49.830872 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:49.830929 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:49.831279 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:49.831390 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:49.831442 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:49.831602 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 02:35:49.831631 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260925 02:35:49.833300 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 32582
I20260925 02:35:49.839390 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 32658
I20260925 02:35:49.845046 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 32733
2026-09-25T02:35:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2818 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:35:49 dist-test-slave-q8mg krb5kdc[483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:35:49 dist-test-slave-q8mg krb5kdc[483](info): set up 2 sockets
Sep 25 02:35:49 dist-test-slave-q8mg krb5kdc[483](info): commencing operation
krb5kdc: starting...
W20260925 02:35:50.895643 10292 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/test-user.keytab.
Sep 25 02:35:50 dist-test-slave-q8mg krb5kdc[483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.13.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
Entry for principal kudu/127.10.13.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab.
I20260925 02:35:51.019531 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40721
--webserver_interface=127.10.13.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:40721,127.10.13.61:41921,127.10.13.60:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:35:51.102996 496 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:35:51.103263 496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:51.103295 496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:51.103338 496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:51.104482 496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:51.104553 496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:51.104578 496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:51.104609 496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:51.104657 496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:51.106238 496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:40721,127.10.13.61:41921,127.10.13.60:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.62.keytab
--principal=kudu/127.10.13.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.62:40721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.13.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:51.106534 496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:51.106776 496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:51.111828 502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:51.111976 496 server_base.cc:1034] running on GCE node
W20260925 02:35:51.111842 504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:51.111924 501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:51.112368 496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:51.112700 496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:51.113865 496 hybrid_clock.cc:648] HybridClock initialized: now 1790303751113839 us; error 46 us; skew 500 ppm
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:51.116384 496 init.cc:377] Logged in from keytab as kudu/127.10.13.62@KRBTEST.COM (short username kudu)
I20260925 02:35:51.116788 496 webserver.cc:460] Webserver started at http://127.10.13.62:42487/ using document root <none> and password file <none>
I20260925 02:35:51.117048 496 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:51.117094 496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:51.117187 496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:51.118240 496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/data/instance:
uuid: "272a62f067884f3d94077ad998738c04"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "936601a1c2c8b22c48ada48b0594a55b"
server_key_iv: "67bbdbddd3cbdf62bbd3bd82a83d5aba"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.118546 496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/wal/instance:
uuid: "272a62f067884f3d94077ad998738c04"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "936601a1c2c8b22c48ada48b0594a55b"
server_key_iv: "67bbdbddd3cbdf62bbd3bd82a83d5aba"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.119935 496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:51.120605 511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:51.120782 496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:51.120918 496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/wal
uuid: "272a62f067884f3d94077ad998738c04"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "936601a1c2c8b22c48ada48b0594a55b"
server_key_iv: "67bbdbddd3cbdf62bbd3bd82a83d5aba"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.120994 496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:51.136108 496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:51.138551 496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:51.138670 496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:51.142730 496 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.62:40721
I20260925 02:35:51.142830 563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.62:40721 every 8 connection(s)
I20260925 02:35:51.143179 496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/data/info.pb
I20260925 02:35:51.144234 564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:51.144713 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 496
I20260925 02:35:51.144825 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-0/wal/instance
I20260925 02:35:51.145061 10292 external_mini_cluster.cc:1428] Setting key b94c2b8be8e2980662878ea12fbe8f71
I20260925 02:35:51.146741 564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.13.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.61@KRBTEST.COM" created.
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:51.159857 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.147532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40721 (local address 127.0.0.1:41672)
0925 02:35:51.147745 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.147785 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.148311 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.148601 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.148604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.148997 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.149161 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.149166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.150181 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.150182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.150848 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.150850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.151374 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.151375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.151442 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.152007 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.152016 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.156021 (+ 4005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.158287 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.158295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.158298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.158494 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.158778 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.158779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.158779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.158867 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.159361 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.159369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.159432 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.159519 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.159651 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260925 02:35:51.159888 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.147552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.10.13.62:40721)
0925 02:35:51.147730 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.147740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.147893 (+ 153us) server_negotiation.cc:407] Connection header received
0925 02:35:51.148485 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.148486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.148503 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.148597 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.149271 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.150081 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.150945 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.151256 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.151390 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.156137 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.156146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.156146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.156153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.158191 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.158583 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.158587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.158587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.158704 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.158938 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.158939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.158939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.159281 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.159378 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.159591 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.159741 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260925 02:35:51.161887 564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:35:51.162308 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.162156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51216)
0925 02:35:51.162212 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.162272 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 02:35:51.162485 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
I20260925 02:35:51.162983 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.162867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51218)
0925 02:35:51.162927 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.162959 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 02:35:51.163120 564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:41921: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab.
Entry for principal kudu/127.10.13.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab.
I20260925 02:35:51.172197 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.61:41921
--webserver_interface=127.10.13.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab
--principal=kudu/127.10.13.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:40721,127.10.13.61:41921,127.10.13.60:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:35:51.212383 564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } attempt: 1
I20260925 02:35:51.213038 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.212834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51232)
0925 02:35:51.212952 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.213005 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 02:35:51.213407 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.213320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51244)
0925 02:35:51.213365 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.213390 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:35:51.213534 564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:41921: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
W20260925 02:35:51.254954 571 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:35:51.255172 571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:51.255189 571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:51.255208 571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:51.256261 571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:51.256295 571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:51.256309 571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:51.256322 571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:51.256335 571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:51.257846 571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:40721,127.10.13.61:41921,127.10.13.60:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.61.keytab
--principal=kudu/127.10.13.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.61:41921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.13.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:51.258044 571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:51.258212 571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:35:51.263731 576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:51.263754 577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:51.263937 571 server_base.cc:1034] running on GCE node
W20260925 02:35:51.263800 579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:51.264330 571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:51.264735 571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:51.265916 571 hybrid_clock.cc:648] HybridClock initialized: now 1790303751265894 us; error 36 us; skew 500 ppm
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:51.269282 571 init.cc:377] Logged in from keytab as kudu/127.10.13.61@KRBTEST.COM (short username kudu)
I20260925 02:35:51.269831 571 webserver.cc:460] Webserver started at http://127.10.13.61:46155/ using document root <none> and password file <none>
I20260925 02:35:51.270109 571 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:51.270174 571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:51.270275 571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:51.271302 571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/data/instance:
uuid: "09ede4a096d84021b22058229d226040"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "6bb7789fe5f3dd4105c1c4193ae367a0"
server_key_iv: "48a7545c130033b40c3ba7fb329b8a00"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.271606 571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/wal/instance:
uuid: "09ede4a096d84021b22058229d226040"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "6bb7789fe5f3dd4105c1c4193ae367a0"
server_key_iv: "48a7545c130033b40c3ba7fb329b8a00"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.272918 571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:35:51.273625 586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:51.273809 571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:35:51.273896 571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/wal
uuid: "09ede4a096d84021b22058229d226040"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "6bb7789fe5f3dd4105c1c4193ae367a0"
server_key_iv: "48a7545c130033b40c3ba7fb329b8a00"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.273952 571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:51.281692 564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } attempt: 2
I20260925 02:35:51.282259 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.282049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51246)
0925 02:35:51.282129 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.282190 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 02:35:51.282591 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.282498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51262)
0925 02:35:51.282543 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.282571 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:35:51.282713 564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.61:41921: Network error: Client connection negotiation failed: client connection to 127.10.13.61:41921: connect: Connection refused (error 111)
I20260925 02:35:51.290726 571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:51.292588 571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:51.292757 571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:51.296731 571 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.61:41921
I20260925 02:35:51.296808 638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.61:41921 every 8 connection(s)
I20260925 02:35:51.297211 571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/data/info.pb
I20260925 02:35:51.298208 639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:51.299746 639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:51.307451 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 571
I20260925 02:35:51.307556 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-1/wal/instance
I20260925 02:35:51.307767 10292 external_mini_cluster.cc:1428] Setting key 419d52b5cfd9f76b2febee3310c94d8a
I20260925 02:35:51.308931 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.300475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.10.13.62:40721)
0925 02:35:51.300537 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.300539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.300756 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:35:51.301347 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.301348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.301366 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.301417 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.302049 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.302538 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.303548 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.303755 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.303838 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.306087 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.306089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.306090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.306097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.307361 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.307830 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.307831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.307832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.307930 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.308197 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.308198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.308199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.308320 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.308405 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.308689 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.308777 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:35:51.309093 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.300427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40721 (local address 127.0.0.1:41682)
0925 02:35:51.300634 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.300672 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.301228 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.301485 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.301489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.301747 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.301949 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.301953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.302639 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.302640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.303445 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.303447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.303859 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.303861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.303992 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.304522 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.304531 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.305988 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.307457 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.307465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.307468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.307752 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.308003 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.308004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.308004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.308123 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.308396 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.308403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.308550 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.308656 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.308816 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260925 02:35:51.309765 639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.13.60@KRBTEST.COM; defaulting to no policy
I20260925 02:35:51.317468 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.310111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51272)
0925 02:35:51.310170 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.310180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.310222 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.310492 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.310493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.310639 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.310685 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.310687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.311561 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.311562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.312116 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.312118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.312594 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.312595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.312660 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.313159 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.313164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.314354 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.316337 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.316338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.316338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.316495 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.316713 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.316713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.316714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.316766 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.317123 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.317123 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.317161 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.317247 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.317351 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:35:51.317535 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.310181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.10.13.61:41921)
0925 02:35:51.310357 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.310362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.310379 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:35:51.310396 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.310398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.310409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.310493 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.310769 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.311461 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.312200 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.312505 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.312596 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.314443 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.314453 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.314453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.314461 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.316261 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.316569 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.316572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.316572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.316625 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.316896 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.316898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.316898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.317048 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.317146 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.317281 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.317417 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260925 02:35:51.318328 639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.13.60@KRBTEST.COM" created.
I20260925 02:35:51.318750 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.318600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42462)
0925 02:35:51.318662 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.318715 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:35:51.318874 589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
I20260925 02:35:51.319345 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.319219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42470)
0925 02:35:51.319288 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.319324 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:35:51.319492 639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:35465: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.13.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab.
Entry for principal kudu/127.10.13.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab.
I20260925 02:35:51.335670 10292 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3SrHiO/build/release/bin/kudu
/tmp/dist-test-task3SrHiO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.60:35465
--webserver_interface=127.10.13.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab
--principal=kudu/127.10.13.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.13.20:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.13.62:40721,127.10.13.61:41921,127.10.13.60:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:35:51.368748 639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } attempt: 1
I20260925 02:35:51.369375 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.369222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42472)
0925 02:35:51.369294 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.369338 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:35:51.369866 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.369679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42484)
0925 02:35:51.369821 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.369846 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260925 02:35:51.370000 639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:35465: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
I20260925 02:35:51.373886 564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } attempt: 3
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:35:51.381412 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.374255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.10.13.61:41921)
0925 02:35:51.374322 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.374325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.374338 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:35:51.374368 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.374369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.374383 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.374438 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.374761 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.375258 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.375993 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.376250 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.376330 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.378638 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.378639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.378640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.378648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.380174 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.380529 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.380531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.380531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.380596 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.380821 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.380822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.380823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.380997 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.381071 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.381244 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.381320 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:35:51.381422 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.374168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51278)
0925 02:35:51.374229 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.374243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.374312 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.374447 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.374448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.374612 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.374679 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.374681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.375364 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.375365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.375906 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.375908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.376329 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.376330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.376428 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.377031 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.377036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.378543 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.380255 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.380256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.380256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.380444 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.380683 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.380684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.380684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.380751 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.381074 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.381076 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.381123 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.381205 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.381312 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 02:35:51.381989 564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:35:51.382423 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.382265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42486)
0925 02:35:51.382317 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.382392 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:35:51.382783 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.382686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42502)
0925 02:35:51.382738 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.382762 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 02:35:51.382908 564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:35465: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
I20260925 02:35:51.414078 564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } attempt: 1
I20260925 02:35:51.414635 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.414480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42518)
0925 02:35:51.414556 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.414604 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:35:51.415117 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.415021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42530)
0925 02:35:51.415072 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.415100 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:35:51.415264 564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:35465: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
W20260925 02:35:51.430341 646 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:35:51.430574 646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:35:51.430593 646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:35:51.430648 646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:35:51.431880 646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:35:51.431924 646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:35:51.431942 646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:35:51.432001 646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:35:51.432025 646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:35:51.433795 646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.13.20:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.13.62:40721,127.10.13.61:41921,127.10.13.60:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/krb5kdc/kudu_127.10.13.60.keytab
--principal=kudu/127.10.13.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.13.60:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.13.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:35:51.434087 646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:35:51.434352 646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:35:51.438158 639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } attempt: 2
I20260925 02:35:51.438757 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.438586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42534)
0925 02:35:51.438677 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.438713 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:35:51.439292 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.439145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42538)
0925 02:35:51.439221 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.439261 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 02:35:51.439427 639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.13.60:35465: Network error: Client connection negotiation failed: client connection to 127.10.13.60:35465: connect: Connection refused (error 111)
W20260925 02:35:51.439657 651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:51.439637 654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:35:51.439708 652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:35:51.440148 646 server_base.cc:1034] running on GCE node
I20260925 02:35:51.440399 646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:35:51.440696 646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:35:51.441911 646 hybrid_clock.cc:648] HybridClock initialized: now 1790303751441889 us; error 34 us; skew 500 ppm
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:35:51.444614 646 init.cc:377] Logged in from keytab as kudu/127.10.13.60@KRBTEST.COM (short username kudu)
I20260925 02:35:51.445178 646 webserver.cc:460] Webserver started at http://127.10.13.60:33499/ using document root <none> and password file <none>
I20260925 02:35:51.445415 646 fs_manager.cc:359] Metadata directory not provided
I20260925 02:35:51.445487 646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:35:51.445649 646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:35:51.446748 646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/data/instance:
uuid: "d30ad760a10a47b19746da8c461a856d"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "f23451d31887747a88c3e183f3ef756d"
server_key_iv: "427f02bf419ad4b7829bf22ca828462d"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.447134 646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/wal/instance:
uuid: "d30ad760a10a47b19746da8c461a856d"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "f23451d31887747a88c3e183f3ef756d"
server_key_iv: "427f02bf419ad4b7829bf22ca828462d"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.448633 646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:35:51.449548 661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:35:51.449746 646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:35:51.449887 646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/data,/tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/wal
uuid: "d30ad760a10a47b19746da8c461a856d"
format_stamp: "Formatted at 2026-09-25 02:35:51 on dist-test-slave-q8mg"
server_key: "f23451d31887747a88c3e183f3ef756d"
server_key_iv: "427f02bf419ad4b7829bf22ca828462d"
server_key_version: "encryptionkey@0"
I20260925 02:35:51.449975 646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:35:51.463608 646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:35:51.465605 646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:35:51.465765 646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:35:51.469889 646 rpc_server.cc:307] RPC server started. Bound to: 127.10.13.60:35465
I20260925 02:35:51.469941 713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.13.60:35465 every 8 connection(s)
I20260925 02:35:51.470320 646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/data/info.pb
I20260925 02:35:51.471323 714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:35:51.471451 10292 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3SrHiO/build/release/bin/kudu as pid 646
I20260925 02:35:51.471570 10292 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3SrHiO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790303478591324-10292-0/minicluster-data/master-2/wal/instance
I20260925 02:35:51.471817 10292 external_mini_cluster.cc:1428] Setting key d81e7bf932ad5e50a2e9cba9d9c55f47
I20260925 02:35:51.473973 714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:51.483877 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.472378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.10.13.60:35465)
0925 02:35:51.472542 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.472550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.472575 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:35:51.473060 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.473067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.473125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.473240 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.474097 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.475368 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.476436 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.477057 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.477164 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.479925 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.479932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.479933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.479938 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.482042 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.482464 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.482467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.482468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.482623 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.482867 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.482868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.482869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.483183 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.483343 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.483473 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.483615 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260925 02:35:51.483922 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.474699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.10.13.62:40721)
0925 02:35:51.474766 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.474768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.474949 (+ 181us) server_negotiation.cc:407] Connection header received
0925 02:35:51.475007 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.475008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.475017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.475066 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.475623 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.476323 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.477440 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.477724 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.477779 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.480546 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.480547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.480547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.480555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.482667 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.483109 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.483110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.483110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.483161 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.483367 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.483368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.483368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.483512 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.483581 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.483780 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.483856 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:35:51.484129 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.474607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.62:40721 (local address 127.0.0.1:41698)
0925 02:35:51.474840 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.474879 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.474948 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.475101 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.475103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.475467 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.475549 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.475550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.476436 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.476437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.477338 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.477340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.477876 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.477878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.477952 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.478390 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.478401 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.480455 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.482753 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.482759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.482761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.483031 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.483231 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.483232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.483233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.483299 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.483590 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.483596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.483642 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.483733 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.483825 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":10,"thread_start_us":115,"threads_started":1}
I20260925 02:35:51.484660 714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.62@KRBTEST.COM
I20260925 02:35:51.490432 564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } attempt: 2
I20260925 02:35:51.492029 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.484956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.61:41921 (local address 127.0.0.1:51288)
0925 02:35:51.485059 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.485068 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.485116 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.485243 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.485244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.485370 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.485402 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.485403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.486130 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.486131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.486692 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.486694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.487109 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.487110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.487170 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.487682 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.487687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.489178 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.490840 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.490841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.490841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.491063 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.491323 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.491324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.491324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.491392 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.491718 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.491719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.491760 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.491844 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.491932 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 02:35:51.492077 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.485015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.10.13.61:41921)
0925 02:35:51.485067 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.485069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.485104 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:35:51.485168 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.485169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.485179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.485240 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.485478 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.486057 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.486773 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.487039 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.487090 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.489265 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.489266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.489267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.489274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.490758 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.491160 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.491161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.491161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.491225 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.491475 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.491476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.491476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.491640 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.491714 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.491889 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.491964 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:35:51.492722 714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:35:51.494500 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.484986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.10.13.62:40721)
0925 02:35:51.485025 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.485027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.485036 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:51.485100 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.485102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.485113 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.485165 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.485716 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.486201 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.486895 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.487084 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.487150 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.490032 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.490034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.490034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.490042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.492289 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.493240 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.493241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.493241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.493307 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.493524 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.493525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.493525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.493686 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.493881 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.494333 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.494409 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.62@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.60@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
I20260925 02:35:51.498198 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.490703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42560)
0925 02:35:51.490753 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.490762 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.490820 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.491270 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.491271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.491407 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.491464 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.491466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.492367 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.492368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.493060 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.493063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.493547 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.493548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.493607 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.494022 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.494027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.495323 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.497123 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.497124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.497125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.497304 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.497523 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.497524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.497525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.497590 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.497873 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.497874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.497925 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.498001 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.498092 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 02:35:51.498523 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.490785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.10.13.60:35465)
0925 02:35:51.491161 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.491163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.491168 (+ 5us) server_negotiation.cc:407] Connection header received
0925 02:35:51.491185 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.491185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.491195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.491265 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.491538 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.492232 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.493157 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.493487 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.493547 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.495417 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.495418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.495419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.495426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.497048 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.497385 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.497386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.497386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.497444 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.497661 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.497662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.497662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.497789 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.497856 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.498340 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.498415 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260925 02:35:51.500057 564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04: Bootstrap starting.
I20260925 02:35:51.500861 564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:51.501262 564 log.cc:824] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:51.501430 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.492993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42572)
0925 02:35:51.493033 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.493042 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.493076 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.493335 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.493336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.493501 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.493533 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.493535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.494830 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.494831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.495357 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.495359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.496095 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.496096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.496152 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.496470 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.496474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.497939 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.500186 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.500186 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.500187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.500399 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.500633 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.500634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.500635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.500695 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.501116 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.501117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.501163 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.501248 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.501340 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.13.61@KRBTEST.COM
I20260925 02:35:51.501772 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.493020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.10.13.60:35465)
0925 02:35:51.493167 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.493168 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.493174 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:35:51.493215 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.493216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.493226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.493289 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.493608 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.494749 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.495456 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.495766 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.495828 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.498016 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.498017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.498018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.498023 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.500090 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.500481 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.500482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.500483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.500549 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.500769 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.500770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.500771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.501028 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.501110 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.501297 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.501456 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260925 02:35:51.502707 564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04: No bootstrap required, opened a new log
I20260925 02:35:51.503325 714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d: Bootstrap starting.
I20260925 02:35:51.504055 714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:51.504158 564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } }
I20260925 02:35:51.504257 564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272a62f067884f3d94077ad998738c04, State: Initialized, Role: FOLLOWER
I20260925 02:35:51.504314 564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } }
I20260925 02:35:51.504374 714 log.cc:824] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:51.504545 721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.504621 564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:51.504630 721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:51.505012 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.496119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.10.13.61:41921)
0925 02:35:51.496158 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.496160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.496622 (+ 462us) server_negotiation.cc:407] Connection header received
0925 02:35:51.496669 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.496670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.496682 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.496743 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.497412 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.498081 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.499041 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.499256 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.499331 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.501943 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.501944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.501945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.501952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.503599 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.504007 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.504008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.504008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.504066 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.504293 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.504294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.504294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.504448 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.504596 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.504832 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.504943 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:35:51.505242 714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d: No bootstrap required, opened a new log
W20260925 02:35:51.506646 733 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 02:35:51.506898 10292 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 02:35:51.506704 733 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:51.507537 714 raft_consensus.cc:348] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } }
I20260925 02:35:51.507683 714 raft_consensus.cc:729] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d30ad760a10a47b19746da8c461a856d, State: Initialized, Role: FOLLOWER
I20260925 02:35:51.507738 538 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "272a62f067884f3d94077ad998738c04"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41702
I20260925 02:35:51.507759 714 consensus_queue.cc:260] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } }
I20260925 02:35:51.507845 538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:35:51.507886 538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:51.508054 714 sys_catalog.cc:564] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:51.508921 538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } }
I20260925 02:35:51.509148 538 leader_election.cc:288] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [CANDIDATE]: Term 1 election: Requested vote from peers 09ede4a096d84021b22058229d226040 (127.10.13.61:41921), d30ad760a10a47b19746da8c461a856d (127.10.13.60:35465)
I20260925 02:35:51.507678 496 master_runner.cc:186] Error getting master registration for 127.10.13.61:41921: OK, instance_id { permanent_uuid: "09ede4a096d84021b22058229d226040" instance_seqno: 1790303751295627 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:35:51.510211 496 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:35:51.509455 688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "272a62f067884f3d94077ad998738c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d30ad760a10a47b19746da8c461a856d"
I20260925 02:35:51.510550 735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.510581 688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:51.510623 735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:51.511445 688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272a62f067884f3d94077ad998738c04 in term 1.
I20260925 02:35:51.511653 514 leader_election.cc:302] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272a62f067884f3d94077ad998738c04, d30ad760a10a47b19746da8c461a856d; no voters:
I20260925 02:35:51.511808 721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [term 1 FOLLOWER]: Leader election won for term 1
W20260925 02:35:51.510211 746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:35:51.511879 746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:51.511967 721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [term 1 LEADER]: Becoming Leader. State: Replica: 272a62f067884f3d94077ad998738c04, State: Running, Role: LEADER
I20260925 02:35:51.512053 721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } }
I20260925 02:35:51.512519 721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272a62f067884f3d94077ad998738c04. Latest consensus state: current_term: 1 leader_uuid: "272a62f067884f3d94077ad998738c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.512595 721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:51.512428 646 master_runner.cc:186] Error getting master registration for 127.10.13.61:41921: OK, instance_id { permanent_uuid: "09ede4a096d84021b22058229d226040" instance_seqno: 1790303751295627 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:35:51.514514 646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:35:51.521798 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.512200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.10.13.62:40721)
0925 02:35:51.512242 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.512244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.513487 (+ 1243us) server_negotiation.cc:407] Connection header received
0925 02:35:51.513569 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.513570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.513583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.513643 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.514761 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.515484 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.516524 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.516809 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.516889 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.518476 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.518477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.518478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.518485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.520121 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.520525 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.520526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.520527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.520585 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.520798 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.520799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.520799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.520946 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.521557 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.521606 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.521685 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:35:51.529112 755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:35:51.529397 755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:35:51.529420 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.513181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.10.13.61:41921)
0925 02:35:51.513229 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.513231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.521744 (+ 8513us) server_negotiation.cc:407] Connection header received
0925 02:35:51.521899 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.521901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.521928 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.521958 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.522845 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.523558 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.524515 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.524844 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.524972 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.526336 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.526337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.526337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.526344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.527999 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.528383 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.528384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.528384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.528452 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.528642 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.528643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.528643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.528798 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.528899 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.529140 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.529315 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:35:51.530520 639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } attempt: 3
I20260925 02:35:51.534109 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.512687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.10.13.60:35465)
0925 02:35:51.512722 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.512724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.521086 (+ 8362us) server_negotiation.cc:407] Connection header received
0925 02:35:51.521189 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.521190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.521214 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.521244 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.522004 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.522745 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.528987 (+ 6242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.529317 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.529379 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.531170 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.531172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.531172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.531180 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.532854 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.533240 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.533241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.533241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.533296 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.533476 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.533477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.533477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.533598 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.533704 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.533929 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.534013 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 02:35:51 dist-test-slave-q8mg krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303751, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.13.61@KRBTEST.COM for kudu/127.10.13.60@KRBTEST.COM
I20260925 02:35:51.536257 688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [term 1 FOLLOWER]: Refusing update from remote peer 272a62f067884f3d94077ad998738c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:51.536576 752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:51.538007 735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272a62f067884f3d94077ad998738c04. Latest consensus state: current_term: 1 leader_uuid: "272a62f067884f3d94077ad998738c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.538272 735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:51.541066 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.530784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.13.60:35465 (local address 127.0.0.1:42592)
0925 02:35:51.530833 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 02:35:51.530843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:35:51.530888 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:35:51.531076 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:35:51.531077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:35:51.531243 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:35:51.531297 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.531299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.532796 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.532797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.533637 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:35:51.533640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.534617 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.534618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:35:51.534672 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.535050 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:35:51.535055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:35:51.536633 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:35:51.539914 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.539915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.539915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.540137 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.540349 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:35:51.540350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:35:51.540350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:35:51.540428 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:35:51.540695 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:35:51.540696 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:35:51.540740 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:35:51.540843 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:35:51.540967 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 02:35:51.541066 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:51.530842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.10.13.60:35465)
0925 02:35:51.530885 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.530887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.530895 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:35:51.530959 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.530960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.530970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.531386 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.531467 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.532675 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.533759 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:51.534495 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:51.534830 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:51.536828 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:35:51.536830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:35:51.536830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:35:51.536837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:35:51.539825 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.540221 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.540222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.540223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.540284 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:35:51.540486 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:35:51.540487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:35:51.540487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:35:51.540633 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:35:51.540693 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:51.540911 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 02:35:51.540981 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:35:51.543175 639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040: Bootstrap starting.
I20260925 02:35:51.543984 639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040: Neither blocks nor log segments found. Creating new log.
I20260925 02:35:51.544303 639 log.cc:824] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040: Log is configured to *not* fsync() on all Append() calls
I20260925 02:35:51.545285 639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040: No bootstrap required, opened a new log
I20260925 02:35:51.546463 721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "272a62f067884f3d94077ad998738c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.546562 721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:51.547391 735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "272a62f067884f3d94077ad998738c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.547479 735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:51.547571 639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } }
I20260925 02:35:51.547710 639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ede4a096d84021b22058229d226040, State: Initialized, Role: FOLLOWER
I20260925 02:35:51.547837 639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } }
I20260925 02:35:51.547895 758 mvcc.cc:205] Tried to move back new op lower bound from 7333084166289256448 to 7333084166194225152. Current Snapshot: MvccSnapshot[applied={T|T < 7333084166289256448}]
I20260925 02:35:51.548192 639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:35:51.548231 613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "272a62f067884f3d94077ad998738c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "09ede4a096d84021b22058229d226040"
I20260925 02:35:51.548327 613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:35:51.548933 759 mvcc.cc:205] Tried to move back new op lower bound from 7333084166289256448 to 7333084166194225152. Current Snapshot: MvccSnapshot[applied={T|T < 7333084166289256448}]
I20260925 02:35:51.549505 613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272a62f067884f3d94077ad998738c04 in term 1.
I20260925 02:35:51.549659 612 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [term 1 FOLLOWER]: Refusing update from remote peer 272a62f067884f3d94077ad998738c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:35:51.550036 752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:35:51.553366 755 catalog_manager.cc:1347] Generated new cluster ID: bce6448f411a4e5e89717dea26373346
I20260925 02:35:51.553421 755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:35:51.554036 777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "272a62f067884f3d94077ad998738c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.554154 777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:51.554214 777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272a62f067884f3d94077ad998738c04. Latest consensus state: current_term: 1 leader_uuid: "272a62f067884f3d94077ad998738c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.554265 777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:51.554301 777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "272a62f067884f3d94077ad998738c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.554349 777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:35:51.554483 778 mvcc.cc:205] Tried to move back new op lower bound from 7333084166289256448 to 7333084166194225152. Current Snapshot: MvccSnapshot[applied={T|T < 7333084166289256448}]
I20260925 02:35:51.558764 752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "272a62f067884f3d94077ad998738c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272a62f067884f3d94077ad998738c04" member_type: VOTER last_known_addr { host: "127.10.13.62" port: 40721 } } peers { permanent_uuid: "09ede4a096d84021b22058229d226040" member_type: VOTER last_known_addr { host: "127.10.13.61" port: 41921 } } peers { permanent_uuid: "d30ad760a10a47b19746da8c461a856d" member_type: VOTER last_known_addr { host: "127.10.13.60" port: 35465 } } }
I20260925 02:35:51.558857 752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04 [sys.catalog]: This master's current role is: LEADER
I20260925 02:35:51.565892 776 catalog_manager.cc:1259] Loaded cluster ID: bce6448f411a4e5e89717dea26373346
I20260925 02:35:51.565958 776 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040: loading cluster ID for follower catalog manager: success
W20260925 02:35:51.566054 776 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 02:35:51.566085 776 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:35:51.569428 755 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:35:51.570207 755 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:35:51.583837 755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 272a62f067884f3d94077ad998738c04: Generated new TSK 0
I20260925 02:35:51.584084 755 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:35:51.612059 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.610189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.10.13.61:41921)
0925 02:35:51.610247 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.610249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.611302 (+ 1053us) server_negotiation.cc:407] Connection header received
0925 02:35:51.611419 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.611420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.611432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.611465 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.611982 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51326: BlockingRecv error: recv got EOF from 127.0.0.1:51326 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 02:35:51.612926 717 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.610473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.10.13.60:35465)
0925 02:35:51.610517 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.610519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.612320 (+ 1801us) server_negotiation.cc:407] Connection header received
0925 02:35:51.612427 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.612428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.612439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.612470 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.612809 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42602: BlockingRecv error: recv got EOF from 127.0.0.1:42602 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 02:35:51.613694 567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.610697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.10.13.62:40721)
0925 02:35:51.610738 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.610740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.613158 (+ 2418us) server_negotiation.cc:407] Connection header received
0925 02:35:51.613261 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.613262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.613273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.613303 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.613612 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41730: BlockingRecv error: recv got EOF from 127.0.0.1:41730 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 02:35:51.616287 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.615394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.10.13.61:41921)
0925 02:35:51.615431 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.615433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.615712 (+ 279us) server_negotiation.cc:407] Connection header received
0925 02:35:51.615834 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.615836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.615870 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.615903 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.616239 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51338: BlockingRecv error: recv got EOF from 127.0.0.1:51338 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 02:35:51.619267 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.618479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.10.13.61:41921)
0925 02:35:51.618512 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.618514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.618776 (+ 262us) server_negotiation.cc:407] Connection header received
0925 02:35:51.618886 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.618887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.618898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.618928 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.619224 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51350: BlockingRecv error: recv got EOF from 127.0.0.1:51350 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 02:35:51.623270 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.622452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.10.13.61:41921)
0925 02:35:51.622487 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.622489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.622707 (+ 218us) server_negotiation.cc:407] Connection header received
0925 02:35:51.622807 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.622807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.622819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.622848 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.623177 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51362: BlockingRecv error: recv got EOF from 127.0.0.1:51362 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 02:35:51.629767 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.628949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.10.13.61:41921)
0925 02:35:51.628986 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.628988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.629221 (+ 233us) server_negotiation.cc:407] Connection header received
0925 02:35:51.629325 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.629326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.629337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.629369 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.629719 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51368: BlockingRecv error: recv got EOF from 127.0.0.1:51368 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 02:35:51.640292 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.639094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.10.13.61:41921)
0925 02:35:51.639135 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.639137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.639672 (+ 535us) server_negotiation.cc:407] Connection header received
0925 02:35:51.639782 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.639783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.639795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.639826 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.640234 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51380: BlockingRecv error: recv got EOF from 127.0.0.1:51380 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 02:35:51.658764 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.657782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.10.13.61:41921)
0925 02:35:51.657831 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.657833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.658121 (+ 288us) server_negotiation.cc:407] Connection header received
0925 02:35:51.658233 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.658234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.658248 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.658279 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.658696 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51394: BlockingRecv error: recv got EOF from 127.0.0.1:51394 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 02:35:51.693024 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.692189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.10.13.61:41921)
0925 02:35:51.692239 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.692242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.692403 (+ 161us) server_negotiation.cc:407] Connection header received
0925 02:35:51.692488 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.692490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.692510 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.692571 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.692949 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51404: BlockingRecv error: recv got EOF from 127.0.0.1:51404 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 02:35:51.758854 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.758161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.10.13.61:41921)
0925 02:35:51.758238 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.758240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.758361 (+ 121us) server_negotiation.cc:407] Connection header received
0925 02:35:51.758434 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.758435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.758446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.758492 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.758805 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51416: BlockingRecv error: recv got EOF from 127.0.0.1:51416 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 02:35:51.888871 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:51.888178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.10.13.61:41921)
0925 02:35:51.888250 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:51.888253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:51.888353 (+ 100us) server_negotiation.cc:407] Connection header received
0925 02:35:51.888398 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:51.888399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:51.888410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:51.888475 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:51.888787 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51418: BlockingRecv error: recv got EOF from 127.0.0.1:51418 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 02:35:52.147001 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:35:52.146261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.10.13.61:41921)
0925 02:35:52.146331 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:52.146333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:52.146345 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:35:52.146421 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:52.146422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:52.146433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:52.146498 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:35:52.146934 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51430: BlockingRecv error: recv got EOF from 127.0.0.1:51430 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:35:52.512373 746 catalog_manager.cc:1259] Loaded cluster ID: bce6448f411a4e5e89717dea26373346
I20260925 02:35:52.512447 746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d: loading cluster ID for follower catalog manager: success
I20260925 02:35:52.513351 746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d: acquiring CA information for follower catalog manager: success
I20260925 02:35:52.513641 746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d30ad760a10a47b19746da8c461a856d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:35:52.567267 776 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040: acquiring CA information for follower catalog manager: success
I20260925 02:35:52.567589 776 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09ede4a096d84021b22058229d226040: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:35:52.663129 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:52.660276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.10.13.61:41921)
0925 02:35:52.660411 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:52.660413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:52.660447 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:35:52.660553 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:52.660554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:52.660620 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:52.660709 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:52.660934 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.661488 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.662309 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.662505 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.662581 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:52.662774 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:52.662886 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:52.662975 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:52.662999 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 02:35:52.663022 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260925 02:35:52.667315 854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:52.664999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.10.13.60:35465)
0925 02:35:52.665121 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:52.665123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:52.665132 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:52.665180 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:52.665180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:52.665217 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:52.665285 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:52.665383 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.665917 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.666605 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.666823 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.666872 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:52.666988 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:52.667088 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:52.667140 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:52.667175 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 02:35:52.667192 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260925 02:35:52.672118 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:52.669128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.10.13.61:41921)
0925 02:35:52.669165 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:52.669167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:52.669219 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:35:52.669357 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:52.669359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:52.669369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:52.669424 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:52.669561 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.670319 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.671311 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.671626 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.671704 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:52.671832 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:52.671868 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:52.671941 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:52.672013 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:35:52.672039 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260925 02:35:52.672120 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:52.669177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.10.13.62:40721)
0925 02:35:52.669331 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:52.669332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:52.669341 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:35:52.669392 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:52.669392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:52.669448 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:52.669526 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:52.669693 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.670428 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.671171 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.671470 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.671534 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:52.671748 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:52.671860 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:52.671915 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:52.672015 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:35:52.672041 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":59,"threads_started":1}
I20260925 02:35:52.672601 854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:35:52.669120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.10.13.60:35465)
0925 02:35:52.669276 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 02:35:52.669278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:35:52.669411 (+ 133us) server_negotiation.cc:407] Connection header received
0925 02:35:52.669527 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:35:52.669529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:35:52.669540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:35:52.669600 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:35:52.669726 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.670201 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.671744 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:35:52.672035 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:35:52.672098 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:35:52.672279 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:35:52.672327 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:35:52.672359 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 02:35:52.672487 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:35:52.672517 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260925 02:35:52.674047 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 496
I20260925 02:35:52.680167 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 571
I20260925 02:35:52.685549 10292 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3SrHiO/build/release/bin/kudu with pid 646
2026-09-25T02:35:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2840 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5659 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274098 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST